.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_bert.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_auto_examples_plot_bert.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_bert.py:


=====================
BERT sklearn
=====================

In order to use `bert-sklearn <https://github.com/charles9n/bert-sklearn>`_ with HiClass, some of scikit-learns checks need to be disabled.
The reason is that BERT expects text as input for the features, but scikit-learn expects numerical features.
Hence, the checks will fail.
To disable scikit-learn's checks, we can simply use the parameter :literal:`bert=True` in the constructor of the local hierarchical classifier.

.. GENERATED FROM PYTHON SOURCE LINES 12-38




.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    Building sklearn text classifier...

      0%|          | 0/231508 [00:00<?, ?B/s]
    100%|##########| 231508/231508 [00:00<00:00, 6750190.69B/s]
    Loading bert-base-uncased model...

      0%|          | 0/440473133 [00:00<?, ?B/s]
      1%|          | 3338240/440473133 [00:00<00:13, 33382272.66B/s]
      2%|2         | 10329088/440473133 [00:00<00:07, 54865734.75B/s]
      4%|3         | 16766976/440473133 [00:00<00:07, 58049664.47B/s]
      5%|5         | 23408640/440473133 [00:00<00:11, 35637670.68B/s]
      6%|6         | 27838464/440473133 [00:00<00:14, 28109925.68B/s]
      8%|7         | 34036736/440473133 [00:00<00:11, 35103745.55B/s]
      9%|9         | 40184832/440473133 [00:01<00:10, 38876214.72B/s]
     10%|#         | 44762112/440473133 [00:01<00:10, 36689951.36B/s]
     11%|#1        | 50415616/440473133 [00:01<00:09, 41321760.94B/s]
     13%|#2        | 57077760/440473133 [00:01<00:08, 47648728.54B/s]
     14%|#4        | 62627840/440473133 [00:01<00:07, 49715260.10B/s]
     15%|#5        | 67963904/440473133 [00:01<00:07, 47874069.83B/s]
     17%|#7        | 74927104/440473133 [00:01<00:06, 53732173.42B/s]
     19%|#8        | 81660928/440473133 [00:01<00:06, 57512216.21B/s]
     20%|#9        | 87622656/440473133 [00:01<00:06, 52765464.78B/s]
     21%|##1       | 93112320/440473133 [00:02<00:06, 51571309.16B/s]
     23%|##2       | 100070400/440473133 [00:02<00:06, 56450616.40B/s]
     24%|##4       | 105871360/440473133 [00:02<00:06, 49664864.90B/s]
     26%|##5       | 112943104/440473133 [00:02<00:05, 55101201.18B/s]
     27%|##7       | 119863296/440473133 [00:02<00:05, 58899836.37B/s]
     29%|##8       | 125974528/440473133 [00:02<00:05, 53546110.15B/s]
     30%|###       | 132965376/440473133 [00:02<00:05, 57851992.72B/s]
     32%|###1      | 138973184/440473133 [00:03<00:07, 40461272.07B/s]
     33%|###2      | 144163840/440473133 [00:03<00:06, 42895302.28B/s]
     34%|###4      | 150983680/440473133 [00:03<00:07, 41025742.67B/s]
     36%|###5      | 157922304/440473133 [00:03<00:05, 47228077.54B/s]
     37%|###7      | 163234816/440473133 [00:03<00:06, 40612466.12B/s]
     39%|###8      | 170188800/440473133 [00:03<00:05, 47009915.57B/s]
     40%|###9      | 176148480/440473133 [00:03<00:05, 45942189.31B/s]
     41%|####1     | 182767616/440473133 [00:03<00:05, 50791650.78B/s]
     43%|####2     | 188731392/440473133 [00:04<00:04, 53046302.68B/s]
     44%|####4     | 194367488/440473133 [00:04<00:05, 44758622.98B/s]
     46%|####5     | 201312256/440473133 [00:04<00:05, 46168284.09B/s]
     47%|####7     | 208133120/440473133 [00:04<00:04, 51418974.33B/s]
     49%|####8     | 214589440/440473133 [00:04<00:04, 54755473.60B/s]
     50%|#####     | 220365824/440473133 [00:04<00:04, 45972312.30B/s]
     51%|#####1    | 226476032/440473133 [00:04<00:04, 49122019.25B/s]
     53%|#####2    | 233082880/440473133 [00:04<00:03, 53408223.64B/s]
     54%|#####4    | 239986688/440473133 [00:05<00:03, 57556553.17B/s]
     56%|#####6    | 246964224/440473133 [00:05<00:03, 60913295.44B/s]
     58%|#####7    | 253720576/440473133 [00:05<00:02, 62785477.43B/s]
     59%|#####9    | 260172800/440473133 [00:05<00:03, 57631281.66B/s]
     61%|######    | 267066368/440473133 [00:05<00:02, 59923274.20B/s]
     62%|######2   | 273211392/440473133 [00:05<00:03, 51272411.61B/s]
     64%|######3   | 280196096/440473133 [00:05<00:02, 55947608.40B/s]
     65%|######4   | 286090240/440473133 [00:05<00:02, 54468654.29B/s]
     67%|######6   | 293223424/440473133 [00:05<00:02, 58955350.50B/s]
     68%|######7   | 299320320/440473133 [00:06<00:02, 59375059.43B/s]
     69%|######9   | 305402880/440473133 [00:06<00:02, 53346789.08B/s]
     71%|#######   | 310938624/440473133 [00:06<00:02, 51495093.31B/s]
     72%|#######2  | 317741056/440473133 [00:06<00:02, 55878761.94B/s]
     73%|#######3  | 323483648/440473133 [00:06<00:02, 54235514.57B/s]
     75%|#######4  | 329016320/440473133 [00:06<00:02, 40915595.18B/s]
     76%|#######6  | 335528960/440473133 [00:06<00:02, 42995207.33B/s]
     78%|#######7  | 342350848/440473133 [00:06<00:02, 48826517.70B/s]
     79%|#######9  | 349076480/440473133 [00:07<00:01, 53402011.24B/s]
     81%|########  | 356067328/440473133 [00:07<00:01, 57722136.09B/s]
     82%|########2 | 362186752/440473133 [00:07<00:01, 50397545.35B/s]
     84%|########3 | 369031168/440473133 [00:07<00:01, 54915663.66B/s]
     85%|########5 | 375827456/440473133 [00:07<00:01, 58350812.26B/s]
     87%|########6 | 381963264/440473133 [00:07<00:01, 47144900.29B/s]
     88%|########8 | 388610048/440473133 [00:07<00:01, 51730740.77B/s]
     90%|########9 | 395258880/440473133 [00:07<00:00, 55463077.66B/s]
     91%|#########1| 401855488/440473133 [00:08<00:00, 58252680.28B/s]
     93%|#########2| 408379392/440473133 [00:08<00:00, 60174256.30B/s]
     94%|#########4| 415054848/440473133 [00:08<00:00, 62025317.79B/s]
     96%|#########5| 421441536/440473133 [00:08<00:00, 58324440.17B/s]
     97%|#########7| 427805696/440473133 [00:08<00:00, 55397405.39B/s]
     99%|#########8| 433883136/440473133 [00:08<00:00, 56834749.35B/s]
    100%|#########9| 439685120/440473133 [00:08<00:00, 46365453.34B/s]
    100%|##########| 440473133/440473133 [00:08<00:00, 50182320.12B/s]

      0%|          | 0/433 [00:00<?, ?B/s]
    100%|##########| 433/433 [00:00<00:00, 3320171.17B/s]
    Defaulting to linear classifier/regressor
    Loading Pytorch checkpoint
    You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
    This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
    train data size: 2, validation data size: 0

    Training  :   0%|          | 0/1 [00:00<?, ?it/s]This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
    This overload of add_ is deprecated:
            add_(Number alpha, Tensor other)
    Consider using one of the following signatures instead:
            add_(Tensor other, *, Number alpha = 1) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:1581.)

    Training  :   0%|          | 0/1 [00:04<?, ?it/s, loss=0.749]
    Training  : 100%|##########| 1/1 [00:04<00:00,  4.33s/it, loss=0.749]
    Training  : 100%|##########| 1/1 [00:04<00:00,  4.40s/it, loss=0.749]

    Training  :   0%|          | 0/1 [00:00<?, ?it/s]
    Training  :   0%|          | 0/1 [00:04<?, ?it/s, loss=0.703]
    Training  : 100%|##########| 1/1 [00:04<00:00,  4.44s/it, loss=0.703]
    Training  : 100%|##########| 1/1 [00:04<00:00,  4.64s/it, loss=0.703]

    Training  :   0%|          | 0/1 [00:00<?, ?it/s]
    Training  :   0%|          | 0/1 [00:04<?, ?it/s, loss=0.687]
    Training  : 100%|##########| 1/1 [00:04<00:00,  4.66s/it, loss=0.687]
    Training  : 100%|##########| 1/1 [00:04<00:00,  4.88s/it, loss=0.687]

    Predicting:   0%|          | 0/1 [00:00<?, ?it/s]
    Predicting: 100%|##########| 1/1 [00:00<00:00,  1.52it/s]
    Predicting: 100%|##########| 1/1 [00:00<00:00,  1.23it/s]
    [['Action' 'The Dark Night']
     ['Action' 'Watchmen']]






|

.. code-block:: default

    from bert_sklearn import BertClassifier
    from hiclass import LocalClassifierPerParentNode

    # Define data
    X_train = X_test = [
        "Batman",
        "Rorschach",
    ]
    Y_train = [
        ["Action", "The Dark Night"],
        ["Action", "Watchmen"],
    ]

    # Use BERT for every node
    bert = BertClassifier()
    classifier = LocalClassifierPerParentNode(
        local_classifier=bert,
        bert=True,
    )

    # Train local classifier per node
    classifier.fit(X_train, Y_train)

    # Predict
    predictions = classifier.predict(X_test)
    print(predictions)


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  32.443 seconds)


.. _sphx_glr_download_auto_examples_plot_bert.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_bert.py <plot_bert.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_bert.ipynb <plot_bert.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_