• Reader2Doc now supports hierarchical processing for PDF, Microsoft Word, and Markdown files

• Each extracted element includes:

  • element_id: Unique UUID identifier per element
  • parent_id: References the parent element's ID for logical document structure

• Enables tree-like navigation and contextual understanding of document hierarchy:

  Chapter 1
   ├── Narrative Text A
   ├── Narrative Text B
  Chapter 2
   ├── Paragraph C
  

• Supports advanced use cases including hierarchical retrieval, graph-based indexing, and multi-level document analysis

• Metadata propagation ensures downstream annotators maintain structural relationships

NerDLApproach Training Optimizations

Significant performance improvements for training of NerDLApproach:

• Reduced Memory Usage with BERT based embeddings: Optimized output embeddings allocations, lowering peak memory footprint during training
• Automatic Dataset Caching: When using setEnableMemoryOptimizer(true) with maxEpoch > 1, input datasets are automatically cached to improve training speed
• Graph Metadata Reuse: NerDLGraphChecker now populates TensorFlow graph metadata that NerDLApproach can reuse, reducing redundant computations during training initialization

With all these improvements you can expect up half the memory consumption and training time on RAM constrained environments (when using setEnableMemoryOptimizer(true)). For larger distributed datasets, the effect will be more pronounced.

XML Reader and Reader2Doc Enhancements

• Single Document Output by Default: Reader2Doc now creates single document annotations per file by default, providing more expected behavior when processing large documents

  • Lines are joined by newline character \n by default, configurable via new setJoinString(string) parameter for custom separators
  • Automatically includes specified attribute values in document output

• Improved Tag Handling: XML reader now ignores empty tags without text content, reducing noise in parsed output

• Enhanced content type handling for application/xml documents

• XML Tag Attribute Extraction: New setExtractTagAttributes(attributes: list[str]) parameter enables extraction of XML attribute values. Example:

  <bookstore>
      <book category="children">
          <title lang="en">Harry Potter</title>
          <author>J K. Rowling</author>
          <year>2005</year>
          <price>29.99</price>
      </book>
      <book category="web">
          <title lang="en">Learning XML</title>
          <author>Erik T. Ray</author>
          <year>2003</year>
          <price>39.95</price>
      </book>
  </bookstore>
  

  We can extract category and lang values with the Reader2Doc Config

  reader2doc = Reader2Doc() \
      .setContentType("application/xml") \
      .setContentPath("../src/test/resources/reader/xml/test.xml") \
      .setOutputCol("document") \
      .setExtractTagAttributes(["category", "lang"])
  

  Resulting in

  children
  en
  Harry Potter
  J K. Rowling
  2005
  29.99
  web
  en
  Learning XML
  Erik T. Ray
  2003
  39.95
  

🐛 Bug Fixes

• Colab Environment Setup: Added Java installation to Colab setup script for improved out-of-the-box compatibility

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.1

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.1

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.1</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.1.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.1.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.1.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.1.jar

What's Changed

• https://github.com/JohnSnowLabs/spark-nlp/pull/14681 by @danilojsl
• https://github.com/JohnSnowLabs/spark-nlp/pull/14682 by @AbdullahMubeenAnwar
• https://github.com/JohnSnowLabs/spark-nlp/pull/14685 by @DevinTDHa
• https://github.com/JohnSnowLabs/spark-nlp/pull/14691 by @DevinTDHa

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/6.2.0...6.2.1