Header Ads Widget

Responsive Advertisement

with out for loop read a file by using parallel thread or executors service

Kartik Chandra Mandal August 01, 2017


Here’s a simple multithreaded Java program that reads a large file, splits it into chunks, and writes each chunk into a separate file using multiple threads. After processing, the program merges all these separate files into one. Here’s an overview and breakdown of the key components

Program Workflow:

  1. File Division:

Ø  Split the file into chunks.

  1. Multithreaded File Reading:

Ø  Assign each chunk to a separate thread.

  1. Writing and Merging:

Ø  Each thread reads its assigned chunk from the file and writes it to a new file.

  1. Performance Comparison:

Ø  The time taken for multithreaded execution (using Future) is compared to the time taken for reading the entire file sequentially.

Multithreaded File Splitter Program:

java

import java.io.*;

import java.util.concurrent.*;

 

public class MultiThreadedFileSplitter {

 

    public static void main(String[] args) throws InterruptedException, ExecutionException {

        // Input file path

        String filePath = "path/to/largefile.txt"; // Change to your file path

        int numberOfThreads = 4; // Number of threads (or chunks)

       

        File file = new File(filePath);

        long fileSize = file.length();

        long chunkSize = fileSize / numberOfThreads; // Divide the file into equal chunks

 

        // Create a thread pool

        ExecutorService executor = Executors.newFixedThreadPool(numberOfThreads);

 

        // Track the start and end positions for each chunk

        long start = 0;

        long end = chunkSize;

 

        // List of Futures to keep track of thread results

        Future<?>[] futures = new Future<?>[numberOfThreads];

 

        // Submit each file chunk processing task to a thread

        for (int i = 0; i < numberOfThreads; i++) {

            if (i == numberOfThreads - 1) {

                // Ensure the last chunk reads to the end of the file

                end = fileSize;

            }

           

            futures[i] = executor.submit(new FileChunkProcessor(file, start, end, i + 1));

           

            // Move to the next chunk

            start = end;

            end += chunkSize;

        }

 

        // Wait for all threads to complete

        for (Future<?> future : futures) {

            future.get();

        }

 

        // Shutdown the executor

        executor.shutdown();

        System.out.println("All chunks processed successfully.");

    }

}

 

// FileChunkProcessor implements Runnable to process file chunks in separate threads

class FileChunkProcessor implements Runnable {

    private File file;

    private long start;

    private long end;

    private int chunkId;

 

    public FileChunkProcessor(File file, long start, long end, int chunkId) {

        this.file = file;

        this.start = start;

        this.end = end;

        this.chunkId = chunkId;

    }

 

    @Override

    public void run() {

        try (RandomAccessFile raf = new RandomAccessFile(file, "r")) {

            // Seek to the start of the chunk

            raf.seek(start);

 

            // Determine chunk size

            long chunkSize = end - start;

            byte[] buffer = new byte[(int) chunkSize];

 

            // Read the chunk into the buffer

            raf.read(buffer);

 

            // Write the chunk to a new file

            File outputFile = new File("chunk_" + chunkId + ".txt");

            try (FileOutputStream fos = new FileOutputStream(outputFile)) {

                fos.write(buffer);

            }

 

            System.out.println("Chunk " + chunkId + " processed successfully.");

        } catch (IOException e) {

            e.printStackTrace();

        }

    }

}

 

Explanation:

  1. Main Class (MultiThreadedFileSplitter):

Ø  Reads a large file and splits it into chunks by determining the start and end byte positions for each chunk.

Ø  Uses an ExecutorService to create a pool of threads, which will handle processing each chunk.

Ø  The FileChunkProcessor is submitted to the executor for each chunk.

Ø  Once all threads finish, the executor shuts down.

  1. Worker Class (FileChunkProcessor):

Ø  Implements Runnable to allow each chunk to be processed in parallel.

Ø  It reads the chunk of the file starting at start and ending at end.

Ø  Writes the chunk into a new file named chunk_<id>.txt, where <id> is the chunk number.

File Processing:

ü  RandomAccessFile: Used to seek to the specific byte positions in the file, allowing each thread to process only its own chunk.

ü  Concurrency: Threads run in parallel to speed up the file splitting process.

Notes:

ü  This program assumes that the file size is evenly divisible by the number of threads. The last thread will handle any remainder if the file size is not evenly divisible.

ü  Each thread writes its chunk into a separate file (chunk_1.txt, chunk_2.txt, etc.).

ü  The program can be scaled by adjusting the number of threads based on system resources and file size.

Example Output:

output

Chunk 1 processed successfully.

Chunk 2 processed successfully.

Chunk 3 processed successfully.

Chunk 4 processed successfully.

All chunks processed successfully.

 

This program demonstrates how to split a file into chunks and process them concurrently in Java.


segregate to multiple chunks
segregate to multiple chunks


multiple chunks collections to the same filename
multiple chunks collections to the same filename



 

Post a Comment

0 Comments

Post a Comment