Awesome Library

autodiff

autodiff是一个c++17的库，使用现代先进的技术实现了自动微分,该库可以支持Eigen库(c++).

CoDiPack

CoDiPack 是一个Header only 的库，实现了自动微分，并且可以与MPI结合

Gzip

一个C++的header only的库，用于数据压缩,在编译使用时需要加上-lz 比如：

g++ test.cpp -lz -o test

示例1：压缩string类型

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


#include <gzip/compress.hpp>
#include <gzip/config.hpp>
#include <gzip/decompress.hpp>
#include <gzip/utils.hpp>
#include <gzip/version.hpp>
#include <iostream>
#include <fstream>
#include <string>

using namespace std;

int CompressString() {
  ofstream output("Temp/out.txt", ios::out | ios::binary);
  string data = "";
  for (int i = 0; i < 100; i++) {
    data += "abcdefg";
  }
  const char * pointer = data.data();
  std::size_t size = data.size();
  // Check if compressed. Can check both gzip and zlib.
  bool c = gzip::is_compressed(pointer, size); // false
  // Compress returns a std::string
  std::string compressed_data = gzip::compress(pointer, size);
  output << compressed_data;
  output.close();
  return compressed_data.size();
}

void DecompressString(int len) {
  // Decompress returns a std::string and decodes both zlib and gzip
  ifstream input("Temp/out.txt", ios::in | ios::binary);
  char *compressed_data = new char [len];
  input.read(compressed_data, len);
  std::string decompressed_data = gzip::decompress(compressed_data, len);
  cout << decompressed_data << endl;
}

// All function calls must pass in a pointer of an
// immutable character sequence (aka a string in C) and its size
int main() {
  int len = CompressString();
  cout << len << endl;
  DecompressString(len);
  return 0;
}

cereal

cereal 是 header-only 的序列化的库(C++11).

示例：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


#include <cereal/archives/binary.hpp>
#include <fstream>
#include <sstream>
#include <iostream>

using namespace std;

class MyClass
{
public:
  int x, y;
  float z;
  string name;
  MyClass() {
    x = 0;
    y = 1;
    z = 2;
    name = "my class";
  }
  template <class Archive>                 //Add serialization methods for your classes
  void serialize( Archive & archive)
  {
    archive( x, y, z );
  }
};

void Serialize() {
  // std::stringstream ss; // any stream can be used
  // archive goes out of scope, ensuring all contents are flushed
  ofstream ss("Temp/data.model", ios::out | ios::binary);
  cereal::BinaryOutputArchive oarchive(ss); // Create an output archive
  MyClass m1, m2, m3;
  oarchive(m1, m2, m3); // Write the data to the archive
}

void Unserialize() {
  ifstream ss("Temp/data.model", ios::in | ios::binary);
  cereal::BinaryInputArchive iarchive(ss); // Create an input archive

  MyClass m1, m2, m3;
  iarchive(m1, m2, m3); // Read the data from the archive
  cout << m1.x << m2.y << m3.z  << " " << m3.name << endl;
}

int main()
{
  Serialize();
  Unserialize();
  return 0;
}

libLBFGS

C语言实现的L-BFGS（Limited-memory Broyden-Fletcher-Goldfarb-Shanno）库

LBFGS++

C++语言实现的L-BFGS（Limited-memory Broyden-Fletcher-Goldfarb-Shanno）header-only库

dlib

C++优化的算法库，比较完整需要安装pthread和libx11-dev

安装好之后，即可以通过以下命令对example文件加下的测试文件进行编译：

g++ -std=c++11  -I../  ../dlib/all/source.cpp krr_classification_ex.cpp  -lpthread -lX11 -lnsl

dlib::mat() \\ Eigen 矩阵转换

MAGMA

The MAGMA project aims to develop a dense linear algebra library similar to LAPACK but for heterogeneous/hybrid architectures, starting with current "Multicore+GPU" systems.

Dask

Dask is a flexible library for parallel computing in Python.

Legion

Legion is a data-centric parallel programming system for writing portable high performance programs targeted at distributed heterogeneous architectures.

ARPACK++

ARPACK++ is an object-oriented version of the ARPACK package.

ARPACK

ARPACK is a collection of Fortran77 subroutines designed to solve large scale eigenvalue problems.

Spectra

Spectra stands for Sparse Eigenvalue Computation Toolkit as a Redesigned ARPACK. It is a C++ library for large scale eigenvalue problems, built on top of Eigen, an open source linear algebra library.

swig

SWIG is a compiler that integrates C and C++ with languages including Perl, Python, Tcl, Ruby, PHP, Java, C#, D, Go, Lua,Octave, R, Scheme (Guile, MzScheme/Racket), Scilab, Ocaml.

Convert the c/C++ to another language.

DPLASMA

Distributed Parallel Linear Algebra Software for Multicore Architectures

PLAPACK is a library infrastructure for the parallel implementation of linear algebra algorithms and applications on distributed memory supercomputers such as the Intel Paragon, IBM SP2, Cray T3D/T3E, SGI PowerChallenge, and Convex Exemplar.

DSCPACK

DSCPACK can be used to solve sparse linear systems using direct methods on multiprocessors and networks-of-workstations. This package is suitable for systems where the coefficient matrix is symmetric and sparse. This solver is written in C; it uses MPI for inter-processor communication and the BLAS library for improved cache-performance.

SuperLU

SuperLU is a general purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations. The library is written in C and is callable from either C or Fortran program. It uses MPI, OpenMP and CUDA to support various forms of parallelism.

CVXOPT

CVXOPT is a free software package for convex optimization based on the Python programming language.

SDP

Semidefinite programming (SDP) has become an extremely active subject in the field of optimization/mathematical programming. It consists in minimizing/maximizing a linear objective function restricted to linear constraints on the cone of positive semidefinite symmetric matrices.

Elemental

Elemental is a modern C++ library for distributed-memory dense and sparse-direct linear algebra, conic optimization, and lattice reduction.

SLEPc

SLEPc, the Scalable Library for Eigenvalue Problem Computations, is a software library for the solution of large sparse eigenproblems on parallel computers. It can be used for the solution of linear eigenvalue problems formulated in either standard or generalized form, as well as other related problems such as the singular value decomposition. It can also be used to solve nonlinear eigenvalue problems, either those formulated as polynomial eigenproblems or more general nonlinear problems.

blaze

Blaze is an open-source, high-performance C++ math library for dense and sparse arithmetic. The Blaze library offers:

high performance through the integration of BLAS libraries and manually tuned HPC math kernels
vectorization by SSE, SSE2, SSE3, SSSE3, SSE4, AVX, AVX2, AVX-512, FMA, SVML and SLEEF
parallel execution by OpenMP, HPX, C++11 threads and Boost threads
the intuitive and easy to use API of a domain specific language
unified arithmetic with dense and sparse vectors and matrices
thoroughly tested matrix and vector arithmetic
completely portable, high quality C++ source code

Margo

Margo is a C library that helps develop distributed services based on RPC and RDMA.

brpc

An industrial-grade RPC framework for building reliable and high-performance services.

OpenUCX

Unified Communication X (UCX) is an award winning, optimized production-proven communication framework for modern, high-bandwidth and low-latency networks.

UCX exposes a set of abstract communication primitives that utilize the best of available hardware resources and offloads. These include RDMA (InfiniBand and RoCE), TCP, GPUs, shared memory, and network atomic operations.

ARROW

A cross-language development platform for in-memory analytics

Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.

Parsl

Parsl is a flexible and scalable parallel programming library for Python. Parsl augments Python with simple constructs for encoding parallelism. Developers annotate Python functions to specify opportunities for concurrent execution. These annotated functions, called apps, may represent pure Python functions or calls to external applications. Parsl further allows invocations of these apps, called tasks, to be connected by shared input/output data (e.g., Python objects or files) via which Parsl constructs a dynamic dependency graph of tasks to manage concurrent task execution where possible.

erpc

eRPC is a fast Remote Procedure Call library designed for modern datacenter networks. It runs over both Ethernet and InfiniBand, and performs comparably to RDMA. Currently, eRPC is intended for use in C++ applications running on bare-metal clusters.

boost

Install form source

1
2


./bootstrap.sh --prefix=/opt/boost # install location
sudo ./b2 install --prefix=/opt/boost --with=all

1
2
3


# centos
yum install boost
yum install boost-devel

Mochi project

The Mochi project is a collaboration between Argonne National Laboratory, Los Alamos National Laboratory, Carnegie Mellon University, and the HDF Group. The objective of this project is to explore a software defined storage approach for composing storage services that provides new levels of functionality, performance, and reliability for science applications at extreme scale.

Margo is a C library enabling the development of distributed HPC services. It relies on Mercury for RPC/RDMA, and Argobots for threading/tasking, hidding the complexity of these two libraries under a simple programming model.

Thallium is a C++14 library wrapping Margo and enabling the development of the same sort of services using all the power of modern C++. It is the recommended library for C++ developers. Note that Thallium also provides C++ wrappers to Argobots.

Mercury is Mochi’s underlying RPC/RDMA library. While it is not necessary to understand how to use Mercury itself when developing with Margo or Thallium (which we recommend), we provide a set of tutorials for those who would need to use it directly rather than through higher level libraries.

Argobots is used for threading/tasking in Mochi. Understanding its underlying programming model may not be necessary at first, for simple Margo or Thallium services, but may become useful to optimize performance or customize the scheduling and placement of threads and tasks in a Mochi service.

Simple: well-defined, user-facing API (gRPC)
Secure: automatic TLS with optional client cert authentication
Fast: benchmarked 10,000 writes/sec
Reliable: properly distributed using Raft etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.

Awesome Library

相关文章：