=======

6.4.2

=======

---------------------------

New Features & Enhancements

---------------------------

* Properly close stream and avoid warning #14774

* [SPARKNLP-1384] Add Hugging Face to Spark NLP model conversion notebooks (ONNX, OpenVINO, GGUF) #14775

* [SPARKNLP-1145] Add BM25 lexical retrieval annotator (Approach + Model) #14776

* [SPARKNLP-1395] Updating git notifications and bumping jsl-llama version #14777

* Sparknlp 1396 introduce new sentence detection annotator #14782

=======

6.4.1

=======

---------------------------

New Features & Enhancements

---------------------------

* Adding perfered engine to pretrained and minor fixes #14689

* fix tags #14695

* Update llama_cpp_in_Spark_NLP_LLMEntityExtractor.ipynb #14756

* Add multimodal image support to VectorDBConnector #14760

* updated GetFewShotexamples to handle multiple cases in MedicalLLMEnti… #14762

* [SPARKNLP-1369] Fix cloud pretrained cache handling for large models #14763

* Add LateChunkEmbeddings annotator #14764

* [SPARKNLP-1359] Implement BiEncoderMultimodalEmbeddings #14767

* Update ResourceDownloader.scala #14769

* [SPARKNLP-1378] HTMLReader Default Headers Error #14770

* [SPARKNLP-1385] Add sentenceAwareFiltering param #14772

=======

6.4.0

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1340] Adding instructions for Databricks Volume #14746

* [SPARKNLP-1279] Introduce LLMNer #14735

* [SPARKNLP-1342] Adding support for ODT files to readers #14747

* [SPARKNLP-1342] Adding support for TSV files to readers #14748

* [SPARKNLP-1346] Adding support for Epub files to readers #14749

* [SPARKNLP-1347] Adding Support for Rich Text Format #14750

* [SPARKNLP-1328] Handle HTML/image fetch failures #14752

* [SPARKNLP-1344] Adding DocumentTitleSplitter annotator #14751

=======

6.3.3

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1335] Implement Layout Aligner annotators #14737

* [SPARKNLP-1336] Enhancements to LightPipeline #14734

* [SPARKNLP-1287] vector db connector annotator #14729

* [SPARKNLP-1231] implement modern bert embeddings #14736

* [SPARKNLP-1341] Add MultiColumnAssembler #14743

---------

Bug Fixes

---------

* [SPARKNLP-1334] Updating POI dependency for readers #14727

=======

6.3.2

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1136] JSON Serialization for Features #14722

* [SPARKNLP-1329] Adding image coordinates to metadata for Reader2Image #14725

* [SPARKNLP-1333] Adding ids input for LightPipeline #14726

---------

Bug Fixes

---------

* Fix out of memory error when copying big models to a cloud storage

=======

6.3.1

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1304] Upgrade jsl-llamacpp to llama.cpp b7247 #14719

* [SPARKNLP-1322] Store Structural Position for Tables and Images #14713

* [SPARKNLP-1323] Adding instructions to setup MS Frabric on spark-nlp #14710

---------

Bug Fixes

---------

* [SPARKNLP-1324] handle nonstandard output tensor names better when loading onnx bert models #14711

* Bugfix when resetting cache in ResourceDownloader #14712

* Addition sentence info into DocumentAssembler output in LightPipeline #14714

=======

6.3.0

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1318] Reader2Image Integration with AutoGGUFVisionModel #14705

---------

Bug Fixes

---------

None

=======

6.2.3

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1317] Further NerDL Optimizations (#14699)

* [SPARKNLP-1315] Fix input data type for CamemBertForTokenClassification

---------

Bug Fixes

---------

None

=======

6.2.2

=======

---------------------------

New Features & Enhancements

---------------------------

None

---------

Bug Fixes

---------

* Fixes WordEmbeddings creating duplicate results

* Minor fixes to NerDLApproach logging

=======

6.2.1

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1306] Adding hierarchical support for Readers

* [SPARKNLP-1297] NerDLApproach Optimizations

* XML Reader and Reader2Doc Improvements

---------

Bug Fixes

---------

* Add Java installation to Colab setup script

=======

6.2.0

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1288] AutoGGUF close model #14671

* [SPARKNLP-1283] Add remove thinking flag #14672

* [SPARKNLP-1293] Enhancements EntityRuler and DocumentNormalizer #14674

* [SPARKNLP-1299] Add Hierarchical Element Identification to HTMLReader #14675

---------

Bug Fixes

---------

* [SPARKNLP-1300] RobertaEmbeddings: changing token sequence in warmup test #14677

=======

6.1.5

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1291] Adding support fort input string column on readers #14665

* [SPARKNLP-1292] Adding fault-tolerance support for malformed XML #14666

* [SPARKNLP-1290] Introducing ReaderAssembler Annotator #14668

---------

Bug Fixes

---------

* fix duplicate loading in FeaturesFallbackReader #14667

=======

6.1.4

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1261] Introducing Reader2Image Annotator #14658

---------

Bug Fixes

---------

None

=======

6.1.3

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1286] GGUFRankingFinisher (#14653)

* [SPARKNLP-1282] Adding improvements to Reader2Doc #14656

* [SPARKNLP-1244] NerDLGraphChecker #14657

---------

Bug Fixes

---------

None

=======

6.1.2

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1256] AutoGGUFReranker

---------

Bug Fixes

---------

* Fixed Issues with loading AutoGGUFVision models

=======

6.1.1

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1183] Upgrade openVINO

* [SPARKNLP-1260] Implement Reader2Table Annotator

* [SPARKNLP-1267] Batch Inference for new llama.cpp version

* [SPARKNLP-1268] Multimodal implementation for new llama.cpp version

* [SPARKNLP-1272] AutoGGUFModel can load AutoGGUFVision pretrained models

---------

Bug Fixes

---------

None

=======

6.1.0

=======

---------------------------

New Features & Enhancements

---------------------------

* [SPARKNLP-1189] Introducing Phi4

* [SPARKNLP-1259] Introducing Reader2Doc Annotator

* [SPARKNLP-1194] Upgrade jsl-llamacpp to newest version

---------

Bug Fixes

---------

* Fix HuggingFace_OpenVINO_in_Spark_NLP_Qwen2VL.ipynb

=======

6.0.5

=======

---------------------------

New Features & Enhancements

---------------------------

* Updating support for Microsoft Fabric (SPARKNLP-1215)

* Adding Markdown Reader (SPARKNLP-1213 )

---------

Bug Fixes

---------

* Fix long-standing issues across the Spark NLP example notebooks (SPARKNLP-1166)

=======

6.0.4

=======

----------------

New Features & Enhancements

----------------

* Introducing MiniLMEmbeddings (SPARKNLP-282)

* Introducing DataFrameOptimizer (SPARKNLP-1086)

* Added PDF Reader features (SPARKNLP-1161)

=======

6.0.3

=======

----------------

New Features & Enhancements

----------------

* Introducing E5-V Universal Embeddings (SPARKNLP-1143)

* Enhanced Chunking Strategies and Behavior (SPARKNLP-1125)

* New XML Reader (SPARKNLP-1119)

----------------

Bug Fixes

----------------

* Fixed typo for Excel reader notebook

=======

6.0.2

=======

----------------

New Features & Enhancements

----------------

* Introducing new Vision Language Models:

* InternVL (SPARKNLP-1123)

* Florance-2 (SPARKNLP-1131)

* Introducing Partition and PartitionTransformer for unstructured data processing (SPARKNLP-1174)

----------------

Bug Fixes

----------------

* Adjusted python type annotations for the `AutoGGUFModel` (#14576)

=======

6.0.1

=======

----------------

New Features & Enhancements

----------------

* Introducing new Vision Language Models:

* SmolVLM (SPARKNLP-1115)

* PaliGemma + PaliGemma2 (SPARKNLP-1121)

* Gemma 3 (SPARKNLP-1124)

* Adding additional parameters options to PDF Reader (SPARKNLP-1158)

----------------

Bug Fixes

----------------

* Fixed typo in Python for RoBertaForMultipleChoice (SPARKNLP-1177)

* Fixing typos in various jupyter notebooks

=======

6.0.0

=======

----------------

New Features & Enhancements

----------------

* Introducing new large language models:

* OLMo model support (SPARKNLP-1006)

* Phi 3.5 Vision model support (SPARKNLP-1060)

* LLAVA model support (SPARKNLP-1033)

* CoHere model support (SPARKNLP-1032)

* Qwen2-VL model support (SPARKNLP-1077)

* Llama 3.2 Vision models (SPARKNLP-1078)

* Deepseek Janus model support (SPARKNLP-1088)

* Added LLAVA v1.5 7b quantized model

* Added StarCoder2 3b int8 model

* New MultipleChoice Transformers:

* AlbertForMultipleChoice (SPARKNLP-1105)

* DistilBertForMultipleChoice (SPARKNLP-1106)

* RoBertaForMultipleChoice (SPARKNLP-1107)

* XlmRoBertaForMultipleChoice (SPARKNLP-1108)

* New file format support:

* Excel files reader (SPARKNLP-1102)

* PowerPoint files reader (SPARKNLP-1103)

* PDF reader (SPARKNLP-1098)

* Text reader (SPARKNLP-1113)

* Other improvements:

* AutoGGUFVisionModel for vision model support (SPARKNLP-1079)

* Added Extractor to SparkNLP (SPARKNLP-1109)

* Updated Python and Scala model names

* Improved error handling for AutoGGUF models

----------------

Bug Fixes

----------------

* Fixed typo in MXBAI notebook

========

5.5.3

========

----------------

Bug Fixes & Enhancements

----------------

* BGEEmbeddings: The default pretrained model for BGEEmbeddings has been changed from "bge_base" to "bge_small_en_v1.5". Users relying on the old default will need to explicitly specify "bge_base" in the pretrained method.

* Added useCLSToken parameter to allow users to choose between CLS token pooling and attention-based average pooling for sentence embeddings.

* BGEEmbeddings: BGEEmbeddings now supports a `useCLSToken` parameter, which defaults to True. This affects the embedding calculation strategy. Existing users should verify their usage and potentially set this parameter explicitly.

* Added HasClsTokenProperties in Scala: Introduced the HasClsTokenProperties trait in Scala providing useCLSToken parameter functionality for relevant annotators.

* Fixing wrong padding in attention mask in `MPNet`, `BGE`, `E5`, `Mxbai`, `Nomic`, `SnowFlake`, and `UAE`. This resulted in wrong inference results in some cases and not equal to the ONNX version in transformers/sentence-transformers.

* Various Performance Optimizations: Multiple changes across different models (Albert, Bart, CLIP, CamemBert, ConvNextClassifier, DeBerta, DistilBert, E5, Instructor, MPNet, Mxbai, Nomic, RoBerta, SnowFlake, UAE, ViTClassifier, VisionEncoderDecoder, Wav2Vec2, XlmRoBertaClassification, XlmRoberta) appear to focus on performance improvements and code cleanup, especially related to OpenVINO and ONNX inference. These may lead to faster inference times.

* Fixing Llama3 download issue in Python.

========

5.5.2

========

----------------

New Features & Enhancements

----------------

* OpenVINO Support for Transformers (PR #14408):

Added OpenVINO inference support to a broad range of transformer-based annotators, including DeBertaForQuestionAnswering, DeBertaForSequenceClassification, RoBertaForTokenClassification, XlmRobertaForZeroShotClassification, BartTransformer, GPT2Transformer, and many others.

* BLIPForQuestionAnswering Transformer (PR #14422):

Introduced a new transformer BLIPForQuestionAnswering for image-based question answering tasks. The transformer processes images alongside associated questions to provide relevant answers.

* AutoGGUFEmbeddings Annotator (PR #14433):

Added AutoGGUFEmbeddings to support embeddings from AutoGGUFModels, providing rich sentence embeddings. Includes an end-to-end example notebook for usage.

* HTML Parsing into DataFrame (PR #14449):

Introduced sparknlp.read().html() to parse local or remote HTML files and convert them into structured Spark DataFrames for easier analysis.

* Email Parsing into DataFrame (PR #14455):

Added sparknlp.read().email() method to parse email files into structured DataFrames, enabling scalable analysis of email content. (Note: Dependent on #14449)

* Microsoft Word Document Parsing into DataFrame (PR #14476):

Added a new feature to parse .docx and .doc files into a Spark DataFrame, streamlining the integration of Word documents into NLP pipelines.

* Microsoft Fabric Support (PR #14467):

Introduced support for leveraging Microsoft Fabric for word embeddings storage and retrieval, enhancing scalability and efficiency.

* cuDNN Upgrade Instructions on Databricks (PR #14451):

Added instructions on upgrading cuDNN for GPU inference and cleaned up redundant Databricks installation instructions.

* ChunkEmbeddings Metadata Preservation (PR #14462):

Modified ChunkEmbeddings to preserve the original chunk’s metadata in the resulting embeddings, ensuring richer contextual information is retained.

* Default Names and Languages for Annotators (PR #14469):

Updated default names and language configurations for newly created seq2seq annotators to improve consistency and clarity.

----------------

Bug Fixes

----------------

* Spark Version Errors (PR #14467):

Resolved issues related to long Spark versions when integrating Microsoft Fabric support.

========

5.5.1

========

----------------

New Features & Enhancements

----------------

* `BertForMultipleChoice` Transformer Added. Enhanced BERT’s capabilities to handle multiple-choice tasks such as standardized test questions and survey or quiz automation.

* Integrated New Tasks and Documentation:

* Added support and documentation for the following tasks:

* Automatic Speech Recognition

* Dependency Parsing

* Image Captioning

* Image Classification

* Landing Page

* Question Answering

* Summarization

* Table Question Answering

* Text Classification

* Text Generation

* Text Preprocessing

* Token Classification

* Translation

* Zero-Shot Classification

* Zero-Shot Image Classification

* `PromptAssembler` Annotator Introduced. Introduced a new annotator that constructs prompts for LLMs using a chat template and a sequence of messages. Accepts an array of tuples with roles (“system”, “user”, “assistant”) and message texts. Utilizes llama.cpp as a backend for template parsing, supporting basic template applications.

----------------

Bug Fixes

----------------

* Resolved Pretrained Model Loading Issue on DBFS Systems.

* Fixed a bug where pretrained models were not found when running AutoGGUF model pipelines on Databricks due to incorrect path handling of gguf files.

========

5.5.0

========

----------------

New Features & Enhancements

----------------

* Introduced QWEN2Transformer (#14188)

* Introduced MiniCPM (#14205)

* Introduced NLLB (#14209)

* Implemented Nomic embeddings (#14217)

* Introduced CamemBertForZeroShotClassification annotator (#14354)

* Implemented Mxbai Embeddings (#14355)

* Introduced AlbertForZeroShotClassification (#14361)

* Introduced Phi-3 (#14373)

* Implemented Starcoder2 for causal language modeling (#14358)

* Integrated llama.cpp (#14364)

* Implemented SnowFlake (#14353)

* Introduced ONNX support to vision annotators (#14356)

* Introduced ONNX and OpenVINO support to Missing Annotators (#14359)

* Added OpenVINO install instructions (#14382)

* Exported notebooks for release candidate (#14393)

========

5.4.2

========

----------------

New Features & Enhancements

----------------

* Added demo notebook for Image Classification Annotators

* Added aggressiveMatching parameter to DateMatcher and MultiDateMatcher annotators

* Added aggressiveMatching parameter to DocumentSimilarityRanker annotator

========

5.4.1

========

----------------

New Features & Enhancements

----------------

* Added support for loading duplicate models in Spark NLP, allowing multiple models from the same annotator to be loaded simultaneously.

* Updated the README for better coherence and added new pages to the website.

* Added support for a stop IDs list to halt text generation in Phi, Mistral, and Llama annotators.

----------------

Bug Fixes

----------------

* Fixed the default model names for Phi2 and Mistral AI annotators.

========

5.4.0

========

----------------

New Features & Enhancements

----------------

* Added OpenVINO Runtime integration for various models, enabling enhanced inference performance. (#14246)

* Added Python APIs to incorporate OpenVINO support. (#14242)

* Introduced support for ONNX models and average pooling in ONNX-based annotators. (#14245)

* Implemented MPNet for token classification. (#14244)

* Added support for MistralAI LLM and LLAMA2. (#14243)

* Improved caching mechanisms in Streamlit demos. (#14241)

* Enhanced models' card and README documentation for Models Hub. (#14240)

* Added OpenVINO GPU dependencies. (#14236)

* Locked macOS version for runners and added missing SBT setup. (#14235)

----------------

Bug Fixes

----------------

* Fixed bugs in Colab notebooks. (#14239)

* Resolved issues with BERT backend and broken annotators. (#14238)

* Corrected LLAMA2 position ID and generation bug. (#14237)

========

5.3.3

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introduce UAEEmbeddings for sentence embeddings using Universal AnglE Embedding, aimed at improving semantic textual similarity tasks

* Introduce critical enhancements and optimizations to the processing of the CoNLL-U format for Dependency Parsers training, including enhanced multiword token handling and improved handling of missing uPos values

* Add example notebook for `DocumentCharacterTextSplitter`

* Add example notebook for `DeBertaForZeroShotClassification`

* Add example notebooks for `BGEEmbeddings` and `MPNetEmbeddings`

* Add example notebook for `MPNetForQuestionAnswering`

* Add example notebook for `MPNetForSequenceClassification`

* Implement cache mechanism for `metadata.json`, enhancing efficiency by avoiding unnecessary downloads

----------------

Bug Fixes

----------------

* Address a bug with serializing ONNX models that lack a `.onnx_data` file, ensuring better reliability in model serialization processes

* Delete redundant `Multilingual_Translation_with_M2M100.ipynb` notebook entries

* Fix Colab link for the M2M100 notebook

========

5.3.2

========

----------------

Bug Fixes

----------------

* Fix and add notebooks to import models from Hugging Face

* Add ONNX and TensorFlow notebooks

* Fix XlnetForSeqeunceClassification and added XlnetForTokenClassificaiton

* Rename DistilBertForZeroShotClassification

* Add missing notebooks

* Add MPNetEmbeddings to annotator

* Fix XLMRoBertaForQuestionAnswering, XLMRoBertaForTokenClassification, and XLMRoBertaForSequenceClassification: Reverted the change in tfFile naming that was causing exceptions while loading and saving the models

* Fix documentation for sparknlp.start()

========

5.3.1

========

----------------

Bug Fixes

----------------

* Fix M2M100 not working on the second run (closing the ONNX Session by mistake)

* Fix ONNX models failing in clusters like Databricks

* Fix `ZeroShotNerClassification` issue with NerConverter

* adding colab notebook for M2M100

========

5.3.0

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing Llama-2 and all the models fine-tuned based on this architecutre. This our very first CasualLM annotator in ONNX and it comes with support for quantization in INT4 and INT8 for CPUs.

* **NEW:** Introducing `MPNetForSequenceClassification` annotator for sequence classification tasks. This annotator is based on the MPNet architecture and is designed to classify sequences of text into a set of predefined classes.

* **NEW:** Introducing `MPNetForQuestionAnswering` annotator for question answering tasks. This annotator is based on the MPNet architecture and is designed to answer questions based on a given context.

* **NEW:** Introducing `M2M100` state-of-the-art multilingual translation. M2M100 is a multilingual encoder-decoder (seq-to-seq) model trained for Many-to-Many multilingual translation. The model can directly translate between the 9,900 directions of 100 languages.

* **NEW:** Introducing a new `DeBertaForZeroShotClassification` annotator for zero-shot classification tasks. This annotator is based on the DeBERTa architecture and is designed to classify sequences of text into a set of predefined classes.

* **NEW:** Implement retreival feature in our `DocumentSimilarity`annotator. The new DocumentSimilarity ranker is a powerful tool for ranking documents based on their similarity to a given query document. It is designed to be efficient and scalable, making it ideal for a variety of RAG applications/

* Add ONNNX support for `BertForZeroShotClassification` annotator.

* Add support for in-memory use of `WordEmbeddingsModel` annotator in server-less cluster. We initially introduced in-memory feature for this annotator for users inside Kubernetes cluster without any `HDFS`, however, today it runs without any issue `locally`, Google `Colab`, `Kaggle`, `Databricks`, `AWS EMR`, `GCP`, and `AWS Glue`.

* New Whisper Large and Distil models.

* Update ONNX Runtime to 1.17.0

* Support new Databricks Runtimes of 14.2, 14.3, 14.2 ML, 14.3 ML, 14.2 GPU, 14.3 GPU

* Support new EMR 6.15.0 and 7.0.0 versions

* Add nobteook to fine-tune a BERT for Sentence Embeddings in Hugging Face and import it to Spark NLP

* Add notebook to import BERT for Zero-Shot classification from Hugging Face

* Add notebook to import DeBERTa for Zero-Shot classification from Hugging Face

* Update EntityRuler documentation

* Improve SBT project and resolve warnings (almost!)

----------------

Bug Fixes

----------------

* Fix Spark NLP Configuration's to set `cluster_tmp_dir` on Databricks' DBFS via `spark.jsl.settings.storage.cluster_tmp_dir` https://github.com/JohnSnowLabs/spark-nlp/issues/14129

* Fix score calculation in `RoBertaForQuestionAnswering` annotator https://github.com/JohnSnowLabs/spark-nlp/pull/14147

* Fix optional input col validations https://github.com/JohnSnowLabs/spark-nlp/pull/14153

* Fix notebooks for importing DeBERTa classifiers https://github.com/JohnSnowLabs/spark-nlp/pull/14154

* Fix GPT2 deserialization over the cluster (Databricks) https://github.com/JohnSnowLabs/spark-nlp/pull/14177

========

5.2.3

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing support for ONNX Runtime in XLMRoBertaForTokenClassification annotator

* **NEW:** Introducing support for ONNX Runtime in XLMRoBertaForSequenceClassification annotator

* **NEW:** Introducing support for ONNX Runtime in XLMRoBertaForQuestionAnswering annotator

* Refactoring AWS SDK use in Spark NLP to reduce the overal size of the library. We have dropped the use of `bundle` and started to directly using `S3` SDK. This will also minimize incompatibilities with other libraries that use AWS SDKs

* Add new notebooks to import DeBertaForQuestionAnswering, DebertaForSequenceClassification, and DeBertaForTokenClassification models from HuggingFace

* Add a new `DocumentTokenSplitter` notebook

* Add a new trainig NER notebook by using DeBerta Embeddings

* Add a new trainig text classification notebook by using INSTRUCTOR Embeddings

* Update `RoBertaForTokenClassification` notebook

* Update `RoBertaForSequenceClassification` notebook

* Update `OpenAICompletion` notebook with new `gpt-3.5-turbo-instruct` model

----------------

Bug Fixes

----------------

* Fix `BGEEmbeddings` not downloading in Python

========

5.2.2

========

----------------

Enhancements

----------------

* Update `aws-java-sdk-bundle` dependency to a version without any CVEs

----------------

Bug Fixes

----------------

* Fix the missing `BGEEmbeddings` from annotator in Python

* Add a new BGE notebook to import models into Spark NLP

* Upload the new true `BGE` models to Spark NLP for text embeddings

========

5.2.1

========

----------------

New Features & Enhancements

----------------

* Add support for Spark and PySpark 3.5 major release

* Support Databricks Runtimes of 14.0, 14.1, 14.2, 14.0 ML, 14.1 ML, 14.2 ML, 14.0 GPU, 14.1 GPU, and 14.2 GPU

* **NEW:** Introducing the `BGEEmbeddings` annotator for Spark NLP. This annotator enables the integration of `BGE` models, based on the BERT architecture, into Spark NLP. The `BGEEmbeddings` annotator is designed for generating dense vectors suitable for a variety of applications, including `retrieval`, `classification`, `clustering`, and `semantic search`. Additionally, it is compatible with `vector databases` used in `Large Language Models (LLMs)`.

* **NEW:** Introducing support for ONNX Runtime in DeBertaForTokenClassification annotator

* **NEW:** Introducing support for ONNX Runtime in DeBertaForSequenceClassification annotator

* **NEW:** Introducing support for ONNX Runtime in DeBertaForQuestionAnswering annotator

* Add a new notebook to show how to import any model from `T5` family into Spark NLP with TensorFlow format

* Add a new notebook to show how to import any model from `T5` family into Spark NLP with ONNX format

* Add a new notebook to show how to import any model from `MarianNMT` family into Spark NLP with ONNX format

----------------

Bug Fixes

----------------

* Fix serialization issue in `DocumentTokenSplitter` annotator failing to be saved and loaded in a Pipeline

* Fix serialization issue in `DocumentCharacterTextSplitter` annotator failing to be saved and loaded in a Pipeline

========

5.2.0

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introduceding the `CLIPForZeroShotClassification` for Zero-Shot Image Classification using OpenAI's CLIP models

* **NEW:** Introduceding the `DocumentTokenSplitter` which allows users to split large documents into smaller chunks to be used in RAG with LLM models

* **NEW:** Introducing support for ONNX Runtime in T5Transformer annotator

* **NEW:** Introducing support for ONNX Runtime in MarianTransformer annotator

* **NEW:** Introducing support for ONNX Runtime in BertSentenceEmbeddings annotator

* **NEW:** Introducing support for ONNX Runtime in XlmRoBertaSentenceEmbeddings annotator

* **NEW:** Introducing support for ONNX Runtime in CamemBertForQuestionAnswering, CamemBertForTokenClassification, and CamemBertForSequenceClassification annotators

* Adding a caching support for newly imported T5 models in TF format to improve the performance to be competitive to ONNX version

* Improve ZIP util and add tests for both ZipArchiveUtil and OnnxWrapper

* Refactor ONNX and add OnnxSession to broadcast

* Update ONNX Runtime to 1.16.3

* Add a new notebook fro structure streaming

----------------

Bug Fixes

----------------

* Fix random dimension mismatch in E5Embeddings and MPNetEmbeddings due to a missing average_pool after last_hidden_state in the output

* Fix batching exception in E5 and MPNet embeddings annotators failing when sentence is used instead of document

* Fix chunk construction when an entity is found

* Fix a bug in library's version in Scala

* Fix Whisper models not downloading due to wrong library's version

* Fix and refactor saving best model based on given metrics during NerDL training

========

5.1.4

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introduceding the `DocumentCharacterTextSplitter` which allows users to split large documents into smaller chunks. `DocumentCharacterTextSplitter` takes a list of separators in order and splits subtexts if they are over the chunk length, considering optional overlap of the chunks.

* **NEW:** Introducing support for ONNX Runtime in RobertaForSequenceClassification annotator

* **NEW:** Introducing support for ONNX Runtime in RobertaForTokenClassification annotator

* **NEW:** Introducing support for ONNX Runtime in RobertaForQuestionAnswering annotator

* Adding an example to load a model directly from Azure using .load() method. This example helps users to understand how to set Spark NLP to load models from Azure

----------------

Bug Fixes

----------------

* Fix a bug with in `Whisper` annotator, that would not allow every model to be imported

* Fix BPE Tokenizer to include a flag whether or not to always prepend a space before words (previous behavior for embeddings)

* Fix BPE Tokenizer to correctly convert and tokenize non-latin and other special characters/words

* Fix `RobertaForQuestionAnswering` to produce the same logits and indexes as the implementation in Transformer library

* Fix the return order of logits in `BertForQuestionAnswering` and `DistilBertForQuestionAnswering` annotators

========

5.1.3

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing support for ONNX Runtime in BertForTokenClassification annotator

* **NEW:** Introducing support for ONNX Runtime in BertForSequenceClassification annotator

* **NEW:** Introducing support for ONNX Runtime in BertForQuestionAnswering annotator

* **NEW:** Introducing support for ONNX Runtime in DistilBertForTokenClassification annotator

* **NEW:** Introducing support for ONNX Runtime in DistilBertForSequenceClassification annotator

* **NEW:** Introducing support for ONNX Runtime in DistilBertForQuestionAnswering annotator

* **NEW:** Setting ONNX configuration such as GPU device id, execution mode, etc. via Spark NLP configs

* Update Whisper documentation with minimum required version of Spark/PySpark (3.4)

----------------

Bug Fixes

----------------

* Fix `module 'sparknlp.annotator' has no attribute 'Token2Chunk'` error in Python when using `Token2Chunk` annotator inside loaded PipelineModel

========

5.1.2

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing VisionEncoderDecoder annotator to generate captions from images

* Add missing enteries in the docs and update them with the new features

* Improve beam search results in BART Transformer

========

5.1.1

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing support for ONNX Runtime in MPNet embedding annotator

* **NEW:** Introducing support for ONNX Runtime in AlbertForTokenClassification annotator

* **NEW:** Introducing support for ONNX Runtime in AlbertForSequenceClassification annotator

* **NEW:** Introducing support for ONNX Runtime in AlbertForQuestionAnswering annotator

* Implement `getVectors` feature in Word2VecModel, Doc2VecModel, and WordEmbeddingsModel annotators. This new feature allows access to the entire tokens and their vectors in the loaded model.

----------------

Bug Fixes

----------------

* Fix how to save and load `Whisper` models

* Fix saving ONNX model on Windows operating system

========

5.1.0

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing WhisperForCTC annotator for Automatic Speech Recognition (ASR)

* **NEW:** Introducing OpenAICompletion and OpenAIEmbeddings annotators

* **NEW:** Introducing MPNet Text Embeddings annotators

* **NEW:** Introducing a new BART for Zero-Shot Text Classification annotator

* **NEW:** Adding ONNX support to E5 Embeddings annotator

* **NEW:** New full support for GCP and Azure distributed storages

* New 150+ MPNet models

* New Databricks 13.3 runtime support

* New EMR 6.12.0 version support

----------------

Bug Fixes

----------------

* Fix max sentence length issue in E5Embeddings

========

5.0.2

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing support for ONNX Runtime in ALBERT, CamemBERT, and XLM-RoBERTa annotators

* **NEW:** Implement ZeroShotNerModel annotator for zero-shot NER based on XLM-RoBERTa architecture

----------------

Bug Fixes

----------------

* Fix MarianTransformers annotator breaking with `java.lang.ClassCastException` in Python

* Fix out of 0.0/1.0 accuracy in SentenceDetectorDL and MultiClassifierDL annotators

* Fix BART issue with low temperature value that only occurred when there are no non infinite logits satisfying the low temperature and top_k values

* Add missing E5Embeddings and InstructorEmbeddings annotators to `annotators` in Scala for easy all-in-one import

========

5.0.1

========

----------------

Bug Fixes & Enhancements

----------------

* Fix `multiLabel` param issue in `XXXForSequenceClassitication` and `XXXForZeroShotClassification` annotators

* Add the missing `threshold` param to all `XXXForSequenceClassitication` in Python

* Fix issue with passing `spark.driver.cores` config as a param into start() function in Python and Scala

* Add new notebooks to export BERT, DistilBERT, RoBERTa, and DeBERTa models to ONNX format

========

5.0.0

========

----------------

New Features & Enhancements

----------------

* **NEW:** Introducing support for ONNX Runtime in Spark NLP. ONNX Runtime is a high-performance inference engine for machine learning models in the ONNX format. ONNX Runtime has proved to considerably increase the performance of inference for many models.

* **NEW:** Introducing **InstructorEmbeddings** annotator in Spark NLP 🚀. `InstructorEmbeddings` can load new state-of-the-art INSTRUCTOR Models inherited from T5 for Text Embeddings.

* **NEW:** Introducing **E5Embeddings** annotator in Spark NLP 🚀. `E5Embeddings` can load new state-of-the-art E5 Models inherited from BERT for Text Embeddings.

* **NEW:** Introducing **DocumentSimilarityRanker** annotator in Spark NLP 🚀. `DocumentSimilarityRanker` is a new annotator that uses LSH techniques present in Spark ML lib to execute approximate nearest neighbours search on top of sentence embeddings, It aims to capture the semantic meaning of a document in a dense, continuous vector space and return it to the ranker search.

----------------

Bug Fixes

----------------

* Fix BART issue with maxInputLength

========

4.4.4

========

----------------

New Features & Enhancements

----------------

* Add `Warmup` stage to loading all Transformers for word embeddings: ALBERT, BERT, CamemBERT, DistilBERT, RoBERTa, XLM-RoBERTa, and XLNet. This helps reducing the first inference time and also validate importing external models from HuggingFace https://github.com/JohnSnowLabs/spark-nlp/pull/13851

* Add new notebooks to import ZeroShot Classifiers for Bert, DistilBERT, and RoBERTa fine-tuned based on NLI datasets https://github.com/JohnSnowLabs/spark-nlp/pull/13845

----------------

Bug Fixes

----------------

* Fix not being able to save models from XXXForSequenceClassification and XXXForZeroShotClassification annotators https://github.com/JohnSnowLabs/spark-nlp/pull/13842

========

4.4.3

========

----------------

New Features & Enhancements

----------------

* New `multilabel` parameter to switch from multi-class to multi-label on all Classifiers in Spark NLP: AlbertForSequenceClassification, BertForSequenceClassification, DeBertaForSequenceClassification, DistilBertForSequenceClassification, LongformerForSequenceClassification, RoBertaForSequenceClassification, XlmRoBertaForSequenceClassification, XlnetForSequenceClassification, BertForZeroShotClassification, DistilBertForZeroShotClassification, and RobertaForZeroShotClassification

* Refactor protected Params and Features to avoid unwanted exceptions during runtime https://github.com/JohnSnowLabs/spark-nlp/pull/13797

* Add proper documentation and instructions for ZeroShot classifiers: BertForZeroShotClassification, DistilBertForZeroShotClassification, and RobertaForZeroShotClassification https://github.com/JohnSnowLabs/spark-nlp/pull/13798

* Extend support for downloading models/pipelines directly by given name or S3 path in ResourceDownloader https://github.com/JohnSnowLabs/spark-nlp/pull/13796

----------------

Bug Fixes

----------------

* Fix pretrained pipelines that stopped working since 4.4.2 release on PySpark 3.0 and 3.1 versions (adding 123 new pipelines were added) https://github.com/JohnSnowLabs/spark-nlp/pull/13805

* Fix pretrained pipelines that stopped working since 4.4.2 release on PySpark 3.2 and 3.3 versions (adding 120 new pipelines) https://github.com/JohnSnowLabs/spark-nlp/pull/13811

* Fix Java compatibility issue caused by SystemUtils dependency https://github.com/JohnSnowLabs/spark-nlp/pull/13806

========

4.4.2

========

----------------

New Features & Enhancements

----------------

* Implement a new Zero-Shot Text Classification for RoBERTa annotator called `RobertaForZeroShotClassification`

* Support Apache Spark 3.4

* Omptize BART models for memory efficiency

* Introducing `cache` feature in BartTransformer

* Improve error handling for max sequence length for transformers in Python

* Improve `MultiDateMatcher` annotator to return multiple dates

----------------

Bug Fixes

----------------

* Fix a bug in Tapas due to exceeding the maximum rank value

* Fix loading Transformer models via loadSavedModel() method from DBFS on Databricks

========

4.4.1

========

----------------

New Features & Enhancements

----------------

* Implement a new Zero-Shot Text Classification for DistilBERT annotator called `DistilBertForZeroShotClassification`

* Adding `threshold` param to `AlbertForSequenceClassification`, `BertForSequenceClassification`, `BertForZeroShotClassification`, `DistilBertForSequenceClassification`, `CamemBertForSequenceClassification`, `DeBertaForSequenceClassification`, LongformerForSequenceClassification`, RoBertaForQuestionAnswering`, `XlmRoBertaForSequenceClassification`, and `XlnetForSequenceClassification` annotators

* Add new notebooks to import models for `SwinForImageClassification` and `ConvNextForImageClassification` annotators for Image Classification

========

4.4.0

========

----------------

New Features

----------------

* Implement a new Zero-Shot Text Classification for BERT annotator called `BertForZeroShotClassification`

* Implement a new ConvNextForImageClassification annotator

* Introducing BART Transformer for text-to-text generation tasks like translation and summarization

* Set custom entity name in Data2Chunk via `setEntityName` param

* Add a new `nerHasNoSchema` param for NerConverter when labels coming from NerDLMOdel and NerCrfModel don't have any schema

----------------

Bug Fixes & Enhancements

----------------

* Fix loading `WordEmbeddingsModel` bug when loading a model from S3 via `cache_folder` config

* Fix `WordEmbeddingsModel` bug failing when it's used with `setEnableInMemoryStorage` set to `True` and LightPipeline

* Remove deprecated parameter enablePatternRegex from EntityRulerApproach & EntityRulerModel

* Deprecate Python 3.6

========

4.3.2

========

----------------

New Features & Enhancements

----------------

* Add S3 support for CoNLL(), POS(), CoNLLU() training classes https://github.com/JohnSnowLabs/spark-nlp/pull/13596

* Add support for non-schema NER (`I-` or `B-`) tags in NerConverter annotator https://github.com/JohnSnowLabs/spark-nlp/pull/13642

* Improve self-hosted examples with better documentation, Docker examples, no broken links, and more https://github.com/JohnSnowLabs/spark-nlp/pull/13575

* Improve error handling for validation evaluation in ClassifierDL and MultiClassifierDL trainable annotators https://github.com/JohnSnowLabs/spark-nlp/pull/13615

----------------

Bug Fixes

----------------

* Fix `Date2Chunk` and `Chunk2Doc` annotators compatibility with PipelineModel https://github.com/JohnSnowLabs/spark-nlp/pull/13609

* Fix `DependencyParserModel` predicting all Chunks as `<no-type>` https://github.com/JohnSnowLabs/spark-nlp/pull/13620

* Removed `calculationsCol` parameter from MultiDocumentAssembler in Python that doesn't actually exist https://github.com/JohnSnowLabs/spark-nlp/pull/13594

========

4.3.1

========

----------------

New Features

----------------

* Easily use external Tokenizers such as spaCy in Spark NLP pipeline

* Implement `params` parameter which can supply custom configurations to the SparkSession

----------------

Bug Fixes & Enhancements

----------------

* Add `entity` field to the metadata in Date2Chunk

* Fix ViT models & pipelines examples in Models Hub

========

4.3.0

========

----------------

New Features

----------------

* Implement HubertForCTC annotator for automatic speech recognition

* Implement SwinForImageClassification annotator for Image Classification

* Introducing CamemBERT for Question Answering annotator

* Implement ZeroShotNerModel annotator for zero-shot NER based on RoBERTa architecture

* Implement Date2Chunk annotator

* Enable params argument in spark_nlp start() function

* Allow doc_id reading CoNLL file datasets

----------------

Bug Fixes & Enhancements

----------------