Here’s a simple multithreaded Java program that reads a
large file, splits it into chunks, and writes each chunk into a separate file
using multiple threads. After processing, the program merges all these separate
files into one. Here’s an overview and breakdown of the key components
Program Workflow:
- File
Division:
Ø
Split the file into chunks.
- Multithreaded
File Reading:
Ø
Assign each chunk to a separate thread.
- Writing
and Merging:
Ø
Each thread reads its assigned chunk from the
file and writes it to a new file.
- Performance
Comparison:
Ø
The time taken for multithreaded execution
(using Future) is compared to the time taken for reading the entire file
sequentially.
Multithreaded File Splitter Program:
java
import
java.io.*; import
java.util.concurrent.*; public class
MultiThreadedFileSplitter { public static void main(String[] args)
throws InterruptedException, ExecutionException { // Input file path String filePath =
"path/to/largefile.txt"; // Change to your file path int numberOfThreads = 4; // Number of
threads (or chunks) File file = new File(filePath); long fileSize = file.length(); long chunkSize = fileSize /
numberOfThreads; // Divide the file into equal chunks // Create a thread pool ExecutorService executor =
Executors.newFixedThreadPool(numberOfThreads); // Track the start and end positions
for each chunk long start = 0; long end = chunkSize; // List of Futures to keep track of
thread results Future<?>[] futures = new
Future<?>[numberOfThreads]; // Submit each file chunk processing
task to a thread for (int i = 0; i <
numberOfThreads; i++) { if (i == numberOfThreads - 1) { // Ensure the last chunk
reads to the end of the file end = fileSize; } futures[i] = executor.submit(new
FileChunkProcessor(file, start, end, i + 1)); // Move to the next chunk start = end; end += chunkSize; } // Wait for all threads to complete for (Future<?> future :
futures) { future.get(); } // Shutdown the executor executor.shutdown(); System.out.println("All chunks
processed successfully."); } } //
FileChunkProcessor implements Runnable to process file chunks in separate
threads class
FileChunkProcessor implements Runnable { private File file; private long start; private long end; private int chunkId; public FileChunkProcessor(File file, long
start, long end, int chunkId) { this.file = file; this.start = start; this.end = end; this.chunkId = chunkId; } @Override public void run() { try (RandomAccessFile raf = new
RandomAccessFile(file, "r")) { // Seek to the start of the chunk raf.seek(start); // Determine chunk size long chunkSize = end - start; byte[] buffer = new byte[(int)
chunkSize]; // Read the chunk into the buffer raf.read(buffer); // Write the chunk to a new file File outputFile = new
File("chunk_" + chunkId + ".txt"); try (FileOutputStream fos = new
FileOutputStream(outputFile)) { fos.write(buffer); } System.out.println("Chunk
" + chunkId + " processed successfully."); } catch (IOException e) { e.printStackTrace(); } } } |
Explanation:
- Main
Class (MultiThreadedFileSplitter):
Ø
Reads a large file and splits it into chunks by
determining the start and end byte positions for each chunk.
Ø
Uses an ExecutorService to create a pool of
threads, which will handle processing each chunk.
Ø
The FileChunkProcessor is submitted to the
executor for each chunk.
Ø
Once all threads finish, the executor shuts
down.
- Worker
Class (FileChunkProcessor):
Ø
Implements Runnable to allow each chunk to be
processed in parallel.
Ø
It reads the chunk of the file starting at start
and ending at end.
Ø
Writes the chunk into a new file named chunk_<id>.txt,
where <id> is the chunk number.
File Processing:
ü RandomAccessFile:
Used to seek to the specific byte positions in the file, allowing each thread
to process only its own chunk.
ü Concurrency:
Threads run in parallel to speed up the file splitting process.
Notes:
ü This
program assumes that the file size is evenly divisible by the number of
threads. The last thread will handle any remainder if the file size is not
evenly divisible.
ü Each
thread writes its chunk into a separate file (chunk_1.txt, chunk_2.txt, etc.).
ü The
program can be scaled by adjusting the number of threads based on system
resources and file size.
Example Output:
output
Chunk 1
processed successfully. Chunk 2
processed successfully. Chunk 3
processed successfully. Chunk 4
processed successfully. All chunks
processed successfully. |
This program demonstrates how to split a file into chunks
and process them concurrently in Java.
segregate to multiple chunks |
0 Comments