object PythonForeachWriter extends Serializable
- Alphabetic
- By Inheritance
- PythonForeachWriter
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- class UnsafeRowBuffer extends Logging
A buffer that is designed for the sole purpose of buffering UnsafeRows in PythonForeachWriter.
A buffer that is designed for the sole purpose of buffering UnsafeRows in PythonForeachWriter. It is designed to be used with only 1 writer thread (i.e. JVM task thread) and only 1 reader thread (i.e. PythonRunner writing thread that reads from the buffer and writes to the Python worker stdin). Adds to the buffer are non-blocking, and reads through the buffer's iterator are blocking, that is, it blocks until new data is available or all data has been added.
Internally, it uses a HybridRowQueue to buffer the rows in a practically unlimited queue across memory and local disk. However, HybridRowQueue is designed to be used only with EvalPythonExec where the reader is always behind the writer, that is, the reader does not try to read n+1 rows if the writer has only written n rows at any point of time. This assumption is not true for PythonForeachWriter where rows may be added at a different rate as they are consumed by the python worker. Hence, to maintain the invariant of the reader being behind the writer while using HybridRowQueue, the buffer does the following - Keeps a count of the rows in the HybridRowQueue - Blocks the buffer's consuming iterator when the count is 0 so that the reader does not try to read more rows than what has been written.
The implementation of the blocking iterator (ReentrantLock, Condition, etc.) has been borrowed from that of ArrayBlockingQueue.
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()