spark-python-data-source
databricks/databricks-agent-skills / experimental/spark-python-data-source
guides building Spark connectors in Python , which improves: connects Spark to any external system
Details
- judge’s score
- 8/10
- place
- outside the packs
- publisher
- databricks
- status
- new
- pinned commit
- c570f88 on GitHub (new tab)
- skill’s last commit
- Jun 2, 2026
- repo stars
- 357
- main type
- Data
Judgement
Very precise triggering and real know-how (serialization of readers, imports on executors, partitioning). Incomplete code snippets (undefined symbols) and truncated annexes: limited precision and quality.
reservations
- Annexes truncated in transmission (production-patterns visible for 931 characters out of 11,635, authentication and error-handling partial): judgment made on the visible part only.
- Several snippets do not run as is: SimpleCommitMessage, YourPartition, _infer_or_return_schema are neither defined nor imported.
- The compatibility metadata (databricks CLI >= 1.0.0) has nothing to do with the PySpark content and is confusing.
- Some examples use datetime.utcnow(), deprecated since Python 3.12.
- Triggering
- 2/2
- Added value
- 2/2
- Precision
- 1/2
- Scope
- 2/2
- Quality
- 1/2
Alerts
1 alert, by category
Data
- medium risk
The examples accept client_secret, password and api_key directly in Spark options, without a reminder to use a secret scope or environment variables.
Original description
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plug…
added on Oct 19, 2026 · license NOASSERTION