Search Results for "parallel compression"
Sort By:
Showing 32 open source projects for "parallel compression"
View related business solutions-
E-commerce Fulfillment For Scaling BrandsFlowspace delivers fulfillment excellence by pairing powerful software and on-the-ground logistics know-how. Our platform provides automation, real-time control, and reliability beyond traditional 3PL capabilities—so you can scale smarter, faster, and easier.
-
Infor M3 ERPEfficiently executing the complex processes of enterprise manufacturers and distributors. Infor M3 is a cloud-based, manufacturing and distribution ERP system that leverages the latest technologies to provide an exceptional user experience and powerful analytics in a multicompany, multicountry, and multisite platform. Infor M3 and related CloudSuite™ industry solutions include industry-leading functionality for the chemical, distribution, equipment, fashion, food and beverage, and industrial manufacturing industries. Staying ahead of the competition means staying agile. Our new capabilities bring improved data-driven insights and streamlined workflows to help you make informed decisions and take quick action.
-
1
compress
Optimized Go Compression Packages
This package provides various compression algorithms. zstandard compression and decompression in pure Go. S2 is a high-performance replacement for Snappy. Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib. snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams. huff0 and FSE implementations for raw entropy encoding. gzhttp Provides client and server wrappers for handling gzipped requests efficiently. pgzip is a separate package that provides a very fast parallel gzip implementation. ... -
2
HDF5
Official HDF5® Library Repository
HDF5 (Hierarchical Data Format v5) is a widely-used data management library and file format for storing large and complex scientific data sets efficiently. -
3
Vortex
The LLVM of columnar file formats
Vortex is a high-performance toolkit designed for working with compressed Apache Arrow arrays, providing functionality for in-memory, on-disk, and over-the-wire data handling. It aims to be an advanced successor to Apache Parquet, offering dramatically faster random access reads and scans, while maintaining similar compression ratios. Vortex's modular design allows for extensibility, enabling developers to implement custom encodings for efficient data management, particularly for large-scale... -
4
Reliable PostgreSQL Backup & Restore
Reliable PostgreSQL backup and restore
pgBackRest aims to be a reliable, easy-to-use backup and restore solution that can seamlessly scale up to the largest databases and workloads by utilizing algorithms that are optimized for database-specific requirements. Compression is usually the bottleneck during backup operations but, even with now-ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with parallel processing. Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. ... -
Intelligent Automation Solutions Built for Modern Finance TeamsDigitally transform your business with workflow automation and integrated payment solutions. Digitally store and secure your data with advanced search and accessibility features that keeps your documents at the tip of your team’s fingers.
-
5
Koog
Koog is the official Kotlin framework for building AI agents
...It features pure Kotlin implementation, seamless Model Control Protocol (MCP) integration for enhanced model management, vector embeddings for semantic search, and a flexible system for creating and extending tools that access external systems and APIs. Ready‑to‑use components address common AI engineering challenges, while intelligent history compression optimizes token usage and preserves context. A powerful streaming API enables real‑time response processing and parallel tool calls. Persistent memory allows agents to retain knowledge across sessions and between agents, and comprehensive tracing facilities provide detailed debugging and monitoring. -
6
Barman
Backup and Recovery Manager for PostgreSQL
Barman (Backup and Recovery Manager) is an enterprise-grade tool for managing backups and disaster recovery of PostgreSQL databases. It supports both full and incremental backups, Point-In-Time Recovery (PITR), and remote backup via SSH. Barman is widely used by DBAs to ensure secure, reliable, and consistent backups in production environments. -
7
FreeRDP
FreeRDP is a free remote desktop protocol library and clients
...Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience. The nightly builds are installed into /opt/freerdp-nightly and can be installed in parallel with the distributions regular freerdp package. Network tracing of RDP can be quite a challenge due to a number of factors such as encryption, compression, and the fact that these protocol features cannot always be disabled or worked around. One of the easiest way of decrypting RDP traffic is to configure the server with a self-signed certificate for which the private key is known, and then use this certificate with a network tracing tool to automatically decrypt the packets. -
8
Google Highway
Performance-portable, length-agnostic SIMD with runtime dispatch
Google Highway is a high-performance C++ library designed to provide portable SIMD (Single Instruction, Multiple Data) vectorization across multiple CPU architectures while maintaining predictable and efficient behavior. It abstracts low-level vector intrinsics into a consistent API that maps closely to hardware instructions, allowing developers to write high-performance code without relying heavily on compiler auto-vectorization. Highway enables the same source code to run across different... -
9
LinDB
LinDB is a scalable, high performance, high availability database
LinDB is a scalable, high-performance, high-availability distributed time series database. A single server could easily support more than one million write TPS; With fundamental techniques like efficient compression storage and parallel computing, LinDB delivers highly optimized query performance. The multi-channel replication protocol supports any amount of nodes, and ensures the system's availability. Schema-free multi-dimensional data model with Metric, Tags, and Fields; The LinQL is flexible yet handy for real-time data analytics. Horizontal scalable is made simple by adding more new broker and storage nodes without too much thinking and manual operations. ... -
Project Planning and Management Software | PlanviewPlanview® ProjectAdvantage (formerly Sciforma) is an enterprise-centric project and portfolio management (PPM) software designed to enable change, drive innovation, and lead in a company's digital transformation. With ProjectAdvantage, teams can strategically track and monitor project data in order to make relevant decisions. It offers multiple features focused on strategic management, functional management, and execution management. A highly scalable and cost-effective solution, ProjectAdvantage is available in various deployment models.
-
10
ClickHouse
A fast open-source OLAP database management system
ClickHouse® is a fast, open-source column-oriented database management system that can generate analytical data reports through SQL queries in real time. According to several independent benchmarks, it far exceeds other comparable column-oriented database management systems, working even up to 1000 times faster. It is able to process hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second. Apart from its blazing speed, ClickHouse is highly... -
11
ERNIE
The official repository for ERNIE 4.5 and ERNIEKit
...The project also emphasizes optimization techniques for large-scale training, including mixed-precision and hybrid-parallel strategies that are commonly needed for multi-node GPU clusters. In addition to training, it includes guidance and example materials intended to help developers adopt ERNIE models for real product scenarios rather than only research demonstrations. -
12
LocustDB
Massively parallel, high performance analytics database
An experimental analytics database aiming to set a new standard for query performance and storage efficiency on commodity hardware. See How to Analyze Billions of Records per Second on a Single Desktop PC and How to Read 100s of Millions of Records per Second from a Single Disk for an overview of current capabilities. Download the latest binary release, which can be run from the command line on most x64 Linux systems, including Windows Subsystem for Linux. When loading .csv or .csv.gz files... -
13
MediaCoder
Universal media transcoding software
MediaCoder is a universal media transcoding software actively developed and maintained since 2005. It puts together most cutting-edge audio/video technologies into an out-of-box transcoding solution with a rich set of adjustable parameters which let you take full control of your transcoding. New features and latest codecs are added or updated constantly. MediaCoder might not be the easiest tool out there, but what matters here is quality and performance. It will be your swiss army knife for... -
14
OpenAS2
AS2 1.1 server implementation in Java
OpenAS2 is a java-based implementation of the EDIINT AS2 standard. It is intended to be used as a server. It is extremely configurable and supports a wide variety of signing and encryption algorithms. Supports very high traffic volume allowing parallel processing of files per partner. SUPPORT: Please use the Github issues and Discussions channels here: https://github.com/OpenAS2/OpenAs2App/ Requirements: - Java 11 or newer (tested with the LTS versions of Java up to 21) ... -
15
ADAMS
ADAMS is a workflow engine for building complex knowledge workflows.
ADAMS is a flexible workflow engine aimed at quickly building and maintaining data-driven, reactive workflows, easily integrated into business processes. Instead of placing operators on a canvas and manually connecting them, a tree structure and flow control operators determine how data is processed (sequentially/parallel). This allows rapid development and easy maintenance of large workflows, with hundreds or thousands of operators. Operators include machine learning (WEKA, MOA, MEKA)... -
16
Sqlite Index Blaster
Create huge Sqlite indexes at breakneck speeds
SQLite Blaster is an advanced SQLite extension that enhances database performance by enabling multi-threading, data compression, and memory optimizations. It is designed for applications that require fast local storage with improved query efficiency. -
17
Crunch
Insane(ly slow but wicked good) PNG image optimization
Crunch is an image compression tool for lossy PNG image file optimization. Using a combination of selective bit depth, color palette reduction and color type, as well as zopfli DEFLATE compression algorithm encoding that employs the pngquant and zopflipng PNG optimization tools, Crunch is effectively able to optimize and compress images with minimal decrease in image quality. While it may produce file size gains larger than those produced by lossless approaches, the impact on image quality... -
18
XZ
Multi-core, cross-platform XZ compression tool written in Go
Parallel implementation of XZ for modern multi-core machines written in Go. This project is licensed under the ISC License. Copyright © 2020-2021 ALBANESE Research Lab Source code: https://github.com/pedroalbanese/xz-2 Visit: http://albanese.atwebpages.com -
19
lzhw
LZHW Windows command line lossless compression tool for tabular files
LZHW Command Line Lossless Compression Tool is a Windows command line tool used to compress and decompress files from and to any form, csv, excel etc without any dependencies or installations. Using an optimized algorithm (LZHW) developed from Lempel-Ziv, Huffman and LZ-Welch algorithms. The tool can work in parallel and most of its code is written in Cython, so it is pretty fast. -
20
ZIPcrypt
Simple Encryption / Decryption Utility
ZIPcrypt is a simple encryption and compression software that uses patented XOTIC 512-bit+, or AES 256-bit encryption. Delivering the best encryption technology on the market today, ZIPcrypt was also designed to be fast, easy to use, and to work in parallel with the SCIFCOM Encryption as a Service website. -
21
Parallel MILC
Parallel decompression for the MILC algorithm
The MILC compression has been developed specifically for medical images and proven to be effective. This algorithm is a parallel version for the decompression phase, meant to exploit the parallel computing potential of the modern hardware. Developed with OpenCL framework (version 1.2), runs on CPU and GPU OpenCL devices. -
22
Icarus Scene Engine
3D .NET engine for Windows, Linux, Mac
Icarus Scene Engine is a cross-platform, CLS-compliant 3D framework, integrating open source APIs into a cohesive all-open-source, all .NET solution. For Windows, MacOSX & Linux. Uses OpenTK, OpenGL, OpenAL APIs for graphics and audio. Runs on Mono/.NET. Links to FFMpeg for streaming video, and FreeType as a temporary conversion mod to bring fonts into a native XML format. Binary builds of Icarus will be infrequent, roughly every 18 months or so, the GIT code updates regularly on a... -
23
COMRAD-MPI
Compression of Large Genomic Datasets using Parallel Computing
COMRAD-MPI is a parallel computing algorithm for reducing the computational time for compressing the large genomic data sets based on COMRAD algorithm. It captures the long range repeat redundancies in large genomes there by providing a way to compress the large DNA data set. Three stages- Substitution, Clean up and Huffman encoding have been parallelized using message passing library. -
24
Parallel DPCM
C# Parallel implementation of DPCM alogrithm
An implementation of the DPCM algorithm for compressing and decompressing an avi video file. It shows the compression size, as it is saved in a txt file near the avi original file. The decompression results in another avi file, saved near the original file. If the width and height of the frames aren't a multiple of 16, they are completed with black pixels. The pixel macroblocks of 16x16 are generated on separate threads for each frame and they are replaced on frames at decompression... -
25
Multi Threaded PDF Merger
Have to merge huge set's of PDF's to a single one, well use this.
...Im Programmer and this wasn't acceptable for me.... This Libary (or Tutorial ?) is the result, its using IText for C# and it's made to use your physical CPU Cores and do Parallel Execution. I will make it short, 1000 PDF needs 15 seconds(On a 6 Core Machine with 50% Usage Peeks), also the Filesize is 50% smaller than Adobe and there is no visual difference (if there change compression...)! This save me a lot of time and since this is for my work also money. Adobe may be great for other tasks but not for simple merging. ...
