ParaMonte: Parallel Monte Carlo and Machine Learning Library

Table of Contents

This is the ParaMonte::Kernel developer and Fortran-user documentation website.

What is ParaMonte?

ParaMonte is a serial / parallel library of Monte Carlo routines for sampling mathematical objective functions of arbitrary-dimensions, in particular, the posterior distributions of Bayesian models in data science, Machine Learning, and scientific inference, with the design goal of unifying the

  • automation of Monte Carlo simulations,
  • user-friendliness of the library,
  • accessibility from multiple programming environments,
  • high-performance at runtime, and,
  • scalability across many parallel processors.

The ParaMonte project repository

The ParaMonte library is open-source is permanently located and maintained on GitHub at:

    https://github.com/cdslaborg/paramonte

ParaMonte usage and examples website

ParaMonte Machine Leaning documentation website

For the developer and Machine Learning Fortran kernel documentation, visit:

    https://cdslaborg.github.io/paramonte-kernel-doc/html

ParaMonte stochastic optimizer documentation website

For information about the samplers usage and examples visit the ParaMonte documentation and examples website at:

    https://www.cdslab.org/paramonte

ParaMonte Samplers

The routines currently supported by the ParaMonte kernel library include:

ParaDRAM

Parallel Delayed-Rejection Adaptive Metropolis-Hastings Markov Chain Monte Carlo Sampler. For a quick start, example scripts, and instructions on how to use he ParaDRAM sampler in your language of choice, visit:

    https://www.cdslab.org/paramonte/notes/usage/paradram/interface

ParaNest

Parallel Nested Sampler and Stochastic Integrator.

ParaMonte Naming conventions

  • The CamelCase naming style is used throughout the entire ParaMonte kernel library.
  • Variable naming conventions.
    • Although the Fortran language is case-insensitive, by convention, all scalar variable names begin with a lower case, whereas all vectors, arrays, types, and module names commonly begin with an upper-case letter. This rule is not strictly imposed on the public interfaces of the library.
    • All constants and parameters should be generally typed upper-case entirely.
      Example usage ⛓
      use iso_fortran_env, only: real64
      integer , parameter :: RK = real64
      real(RK), parameter :: PI = acos(-1._RK)
    • The names of variables that represent vectors of values can be suffixed with Vec or Vector (for example, ProposalStartStdVec, ...) although, this is not recommended since it can conflict with generic programming. Use this naming convention only if the variable is to always represent a vector.
    • The names of variables that represent matrices of values can be suffixed with Mat or Matrix (for example: ProposalStartCorMat, ...) although, this is not recommended since it can conflict with generic programming. Use this naming convention only if the variable is to always represent a matrix.
    • The names of variables that represent lists of varying-size values can be suffixed with List (for example, domainAxesNameList, ...) although, this is not recommended since such naming can interfere with generic programming.

    • Significant attempt has been made to end all logical (boolean) variables with a passive verb, such that the full variable name virtually forms a proposition, that is, an English-language statement that should be either .true. or .false., for example, parallelismMpiFinalizeEnabled.

      Occasionally, names that begin with the verb is can be also used to label logical objects. But as a general rule, names that begin with a verb should be reserved for procedures.

  • Procedure naming conventions.
    • Procedure (whether function, subroutine, or type-bound procedure (class method)) names should be descriptive of the action performed by the procedure. For example,
      • genCovUpp means generate (a) covariance upper(-triangular matrix).
      • symmetrizeMatUpp means symmetrize (the input) matrix upper(-triangular).
    • Procedure names should virtually always begin with a lower-case verb.
    • Function naming conventions.
      • Function names should preferably begin with gen. The reasoning is simple; Functions generate objects of various types and return them. While it is equally possible to use other verbs to prefix function names, like get, such verbs should be reserved for the subroutine equivalents of the same functions. For example, symmetrizeMatUpp is a subroutine that symmetrizes the input upper-triangle matrix that has intent(inout). Equivalently, the function genMatSymFromMatUpp(MatUpp) result(MatSym) generates a symmetric version of the input matrix and returns it as the function result.
        Exceptions to this naming convention are allowed, for example, when a procedure is expected to exist only as a function (and not a subroutine). An example of such an exception is replace.
      • Functions that return objects of type logical should be preferably prefixed with is or be named such that the name begins with a verb and reads as a proposition, evaluating to either .true. or .false..
    • Subroutine naming conventions.
      • The keyword gen should be avoided as a prefix for subroutine names since subroutines do not generate any objects to return. Subroutines only manipulate and transform arguments passed to them.
    • Exceptions to the above rules are allowed when the procedure name is exceptionally famous or it is very inconvenient to prefix the procedure name with a verb.

ParaMonte Abbreviation Guidelines

The following list of abbreviations is in alphabetical order to enable faster search:

  • The abbreviation avg stands for average (rarely used).
  • The abbreviation cdf stands for Cumulative Distribution Function in the context of statistics. Example: getNormCDF().
  • The abbreviation cho stands for Cholesky factorization. Example: getChoLow().
  • The abbreviation cor stands for correlation. Example: genCorUppFromCovUpp().
  • The abbreviation cov stands for covariance. Example: genCovUpp().
  • The abbreviation cum stands for cumulative. Example: genCumSum().
  • The abbreviation coef stands for coefficient. Example: CorCoef().
  • The abbreviation def stands for definite (mostly in procedure names dealing with positive-definite matrices) or default (mostly in internal procedure naming for default values of optional input arguments). If def stands for default to represent the alternative for optional dummy arguments of procedures, then it should be preferably suffixed to the name of the alternate object for the optional argument as _def, all lower-case.
    Example usage ⛓
    function genInvPosDefMat(PosDefMat) result(InvPosDefMat)
    real, allocatable :: InvPosDefMat(:,:) ! Def stands for definite.
    end function genInvPosDefMat
    subroutine assert(assertion,renabled)
    logical, intent(in) :: assertion
    logical, intent(in), optional :: renabled
    logical :: renabled_def ! def stands for default.
    renabled_def = .false.
    if (present(renabled)) renabled_def = renabled
    end subroutine assert
  • The abbreviation den stands for density, mostly in the context of statistical procedures and objects. Example: genLogProbDen().
  • The abbreviation det stands for determinant, mostly in the context of Matrix and linear algebra. Example: getInvPosDefMatSqrtDet().
  • The abbreviation dia stands for diagonal, mostly in the context of Cholesky factorization. Example: ChoDia().
  • The abbreviation desc stands for description, mostly as components of ParaMonte sampler classes and as dummy argument to assert in tests.
  • The abbreviation diag stands for diagonal. Example: genDiagMat().
  • The abbreviation diff stands for difference. Example: getDistSortedExpDiff().
  • The abbreviation dist stands for distance or distribution depending on the context. Example: DistMulti_type.
  • The abbreviation eff stands for effective. Example: effSamSize.
  • The abbreviation exp stands for exponential or exponentiated. Example: getDistSortedExpDiff().
  • The abbreviation gen stands for generate. Example: genDiagMat().
    Remarks
    The prefix gen is commonly used as prefix for the name of function procedures because a function generates and returns an object. This is contrary to the prefix get at the beginning of subroutine procedures that return the result inside an already given object.
  • The abbreviation inv stands for inverse. Example: genInvMat().
  • The abbreviation ks stands for Kolmogorov-Smirnov test. Example: genMahalSq().
  • The abbreviation lin stands for linear. Example: genLinSpace().
  • The abbreviation low stands for lower triangle of a matrix or lower limits. Example: getChoLow().
  • The abbreviation mahal stands for Mahalanobis distance. Example: genMahalSq().
  • The abbreviation mat stands for matrix. Example: genInvMat().
  • The abbreviation multi stands for multivariate mostly used in the context of statistical distributions. Example: DistMulti_type.
  • The abbreviation msn stands for Multivariate Skew-Normal mostly used in the context of the statistical MultiVariate Skew-Normal distribution. Example: genRandMSN().
  • The abbreviation mvn stands for MultiVariate Normal mostly used in the context of the statistical MultiVariate Normal distribution. Example: genRandMVN().
  • The abbreviation mvu stands for MultiVariate Uniform mostly used in the context of the statistical MultiVariate (ellipsoidal) Uniform distribution. Example: getRandMVU().
  • The abbreviation norm stands for normal in the context of statistical distributions or normalization factor. Example: DistMultiNorm_type.
  • The abbreviation normed stands for normalized mostly in the context of statistical samples. Example: NormedSample.
  • The abbreviation pdf stands for Probability Density Function in the context of statistics. Example: genNormLogPDF().
  • The abbreviation pos stands for positive. Example: genInvPosDefMat().
  • The abbreviation prob stands for probability, mostly in the context of statistical applications. Example: genLogProbDen().
  • The abbreviation proc stands for procedure, particularly, when it appears as the suffix _proc in abstract interface definitions.
    Example usage ⛓
    use ParaMonteLogFunc_mod, only: getLogFunc_proc
    procedure(getLogFunc_proc) :: getLogFunc
    ParaMonteLogFunc_mod::getLogFunc_proc
    Return the value of the objective function at the given input Point. This is the abstract C interface...
    Definition: ParaMonteLogFunc_mod.f90:69
    ParaMonteLogFunc_mod
    This module defines the abstract interface of the objective function to be called by ParaMonte routin...
    Definition: ParaMonteLogFunc_mod.f90:53
  • The abbreviation rand stands for random, mostly in the context of statistics. Example: genRandState().
  • The abbreviation ref stands for reference, mostly in the context of testings to represent the reference values for comparison. Example: mean_ref.
  • The abbreviation sam stands for sample, mostly in the context of statistics. Example: effSamSize.
  • The abbreviation sq stands for squared. Example: genMahalSq().
  • The abbreviation stat stands for statistics. Example: StatDRAM_type.
  • The abbreviation std stands for standard deviation. Example: StdVec.
  • The abbreviation sym stands for symmetric. Example: genMatSymFromMatUpp.
  • The abbreviation uni stands for univariate, mostly used in the context of statistical distributions. Example: DistUni_type.
  • The abbreviation upp stands for upper triangle of a matrix or upper limits. Example: getChoUpp().
  • The abbreviation vec stands for vector. Example: StdVec.
  • The abbreviation unif stands for uniform, mostly in the context of the uniform statistical distribution. Example: getRandUnif().

ParaMonte Developer Guidelines and Warnings

  • Case-sensitivity.
    • Although the Fortran programming language is case-INsensitive, please preserve and ensure the case-sensitivity compliance of all object names including module, type, procedure, and variable names in all codes and files. This is crucial for the proper hyper-linking of different HTML pages of the ParaMonte :: Kernel developer library that you are reading right now, that is generated by Doxygen. For example, if the module name is Math_mod, do not write it in any other way like math_mod or MATH_MOD.
    • If a variable or procedure argument in a generic interface can be both scalar and array, it should be preferably capitalized since a scalar can be envisioned as particular case of an array with one element.
  • Clarity vs. conciseness.
    • Always prefer clarity to conciseness, specially when choosing variable, argument, or procedure names.
  • Preprocessor MACRO conventions.
    • MACRO names, if they are to used as conditions in preprocessing directives, should always read like a proposition that evaluates to either true or false. For example, CHECK_ENABLED, IS_WINDOWS.
    • MACRO names, if they are to used as conditions, must be always given a value of 1 if true. This is specially important if the macro is defined from within the source code. Why? Such value assignment eliminates reliance of the conditional construct on the definition status of the macro as the condition. For example, the following conditional construct executes because CHECK_ENABLED is assigned a value of 1.
      Example usage ⛓
      #define CHECK_ENABLED 1
      #if CHECK_ENABLED
      if (.not. present(Err) .and. ChoDia(1) < 0._RK) error stop
      #endif
  • Controlling the runtime checks.
    • The ParaMonte library has two different build modes (for any build type: debug, testing, release). The build mode is determined (by the choice of user) via the preprocessor flag CHECK_ENABLED. If this macro is defined, then certain checks on procedure arguments and validity tests in various modules and procedures of the library are automatically performed at runtime to ensure the validity and accuracy of the calculations. This will however, lower the runtime performance of the library. As such, the ParaMonte build scripts define the preprocessor macro CHECK_ENABLED by default for the debug and testing build types, and undefine the macro for the release build. For any build type, this behavior can be overridden by the user/developer when building the library. Refer to the help file of the build scripts for more details and instructions.
      Warning
      Do NOT lower the case of any uppercase PURE procedure attribute. The uppercase PURE is a macro that expands to pure when runtime checks are disabled, that is, when CHECK_ENABLED is undefined or set to 0, otherwise, it is empty. The worst consequence of failing to pay attention to this matter is that the library will not compile and the developer will have to fix any mistakes they have made with PURE preprocessor macros.
    • When the library is built with tests enabled, a number of fatal error stop statements are disabled in some procedures (like the ParaMonte samplers), in particular, when the library is built in parallel mode. Such temporary disabling of global program halts is needed to ensure the error signals are propagated correctly with the library and across multiple processors. However, such extra testing communications (in particular, in parallel mode) can be very costly and therefore, the library should never be built for production release with any tests of the library enabled.
  • Data hiding and privacy
    • Module entities should be all preferably public unless an entity is guaranteed to never be needed outside the module, in which case, it should be given the private attribute. An example of such guaranteed privacy is a type-bound procedure. The reason for keeping all module entities public is simple. Generic interfaces cannot be passed as dummy arguments. Therefore, all module procedures, even though accessible via generic interfaces, should be kept public.
    • Avoid declaring and using global module variables, unless the variable is guaranteed to remain unchanged after program initialization. If a module variable is only initialized once throughout the entire life of the program, then it should be preferably prefixed with mc_ to indicate the variable is a module runtime constant (not a parameter, since parameters are compile-time constants). in which case, it should be given the private attribute. An example of such guaranteed privacy is a type-bound procedure. The reason for keeping all module entities public is simple. Generic interfaces cannot be passed as dummy arguments. Therefore, all module procedures, even though accessible via generic interfaces, should be kept public.

ParaMonte Kernel Documentation Guidelines

  • Doxygen custom command orderings.
    • The Doxygen tag \brief must always be the first line of the documentation of modules, types, and procedures. Example: Math_mod.
    • The Doxygen tag \details, if it exists, must always immediately follow the Doxygen tag \brief. Example: Math_mod.
    • The Doxygen tag \param, if any number of it exists, must always immediately follow the Doxygen tag \brief (or \details if it exists). Example: genMean().
    • The Doxygen tag \return, must be exclusively used to indicate the return value of functions. If it exists, it must appear immediately after the set of \param tags. Example: genMean().
    • If a generic interface is being documented, the ParaMonte custom command \interface must appear immediately after the Doxygen \return, \param, \details, or \brief tags in the specified order, if any exists.
    • The Doxygen tag \warning, if any number of it exists, must immediately follow the Doxygen tag \return if it exists, otherwise \param if it exists, otherwise \details if it exists, otherwise \brief. The \warning tag must be used to highlight situations that require special attention of the user, otherwise, there is a danger for the code section being documented to not behave normally as one may expect.
    • The Doxygen tag \attention has the same functionality and usage as \warning. Therefore, \warning should be preferred wherever \attention is needed. Exceptions are allowed and if they occur, the same documentation conventions as those of \warning also apply to the tag \attention.
    • The Doxygen tag \remark, if any number of it exists, must immediately follow the Doxygen tag \warning if it exists, otherwise the Doxygen tag \return if it exists, otherwise \param if it exists, otherwise \details if it exists, otherwise \brief. The tag \remark should be reserved for explaining behavior that is directly related to the code segment being documented, but its knowledge is not so critical as warrant the use of a \warning tag.
    • The Doxygen tag \note, if it exists, must appear after all \warning and \attention and \remark tags and immediately before the ParaMonte custom command tag \see if it exists, otherwise immediately before \example for examples (if it exists).
    • The Doxygen tag \see, if it exists, must appear after all \warning and \remark and \note tags. If more than one item for the \see command exists, each must be written on a separate line and each line must end with the HTML line-break tag <br>. Example: See below.
    • If any example exists, it must appear immediately after the \see tag, otherwise after \note, \remark, \warning, \param, \details, or \brief if any exists. ParaMonte examples are initiated by the custom command \example devised in the config.txt file of ParaMonte Doxygen documentation. If the example exists in an external file, then it must be included via the Doxygen \include command, followed immediately by the ParaMonte custom Doxygen command \compile which inserts the generic example compile commands for the example, followed optionally but immediately by the output file of the example inserted in the documentation via the \include command, followed immediately by the inclusion of any other visualization or postprocessing scripts and output. **In all steps, it is imperative to not leave any empty lines between the successive commands of the example section, designated by the \example, otherwise, each empty line will start a new paragraph in the documentation. Example: See below.
    • The Doxygen \test tag, if any exists, must appear immediately after the example section designated by the \example tag. The Doxygen \todo tag, if any exists, must appear immediately after the \test tag or any other tag immediately preceding it. The Doxygen \bug tag, if any exists, must appear immediately after the \todo tag or any other tag immediately preceding it.
    • The closing command of each documentation section must be the ParaMonte custom command \final separated from the tags before and after by an empty line.
    • The Doxygen \author tag is the last command to appear in any documentation section, and it must preferably have the format exemplified in the example below.

  • ParaMonte Doxygen custom commands.
    To simplify documentation and avoid retyping certain frequently used keywords and sentences, a number of Doxygen aliases are predfined in the ParaMonte Doxygen config.txt file. These include (but are not limited to):

    • \warnpure Inserts a \warning about procedures that are impure when the library is build the preprocessor macro CHECK_ENABLED=1.
    • \elemental Inserts a \remark tag indicating that the procedure of interest is elemental.
    • \pure Inserts a \remark tag indicating that the procedure of interest is pure.
    • \interface Starts a Possible calling interfaces paragraph where different calling interfaces of a procedure can be listed.
    • \benchmark Starts a new Benchmark paragraph which is hyper-linked to the generic anchor #benchmark at the same location on the same page.
    • \benchmark{xxx} Starts a new Benchmark paragraph which is hyper-linked to the specific anchor #benchmark-xxx at the same location on the same page.
    • \benchmark{xxx, This is the benchmark title} Starts a new Benchmark paragraph which is hyper-linked to the specific anchor #benchmark-xxx at the same location on the same page with the title This is the benchmark title.
    • \example Starts a new Example usage paragraph which is hyper-linked to the generic anchor #example at the same location on the same page.
    • \example{xxx} Starts a new Example usage paragraph which is hyper-linked to the specific anchor #example-xxx at the same location on the same page.
    • \compile Inserts the set of example compile commands.
    • \output Inserts a title line for the output section of an example paragraph.
    • \postproc Inserts a title line for the postprocessing section of an example paragraph.
    • \abbr Inserts a \remark tag about the naming abbreviations used in the library.
    • \naming Inserts a \remark tag about the naming conventions used in the library.
    • \license Inserts a \remark tag about the generic licensing of the library.
    • \final Inserts the set of final generic remarks that should appear at the end of each documentation section.
    • \RK Inserts a hyper-link reference to the default real kind used in the library.
    • \RK32 Inserts a hyper-link reference to the real32 real kind used in the library.
    • \RK64 Inserts a hyper-link reference to the real64 real kind used in the library.
    • \RK128 Inserts a hyper-link reference to the real128 real kind used in the library.
    • \CK Inserts a hyper-link reference to the default complex kind used in the library.
    • \CK32 Inserts a hyper-link reference to the real32 complex kind used in the library.
    • \CK64 Inserts a hyper-link reference to the real64 complex kind used in the library.
    • \CK128 Inserts a hyper-link reference to the real128 complex kind used in the library.
    • \IK8 Inserts a hyper-link reference to the int8 integer kind used in the library.
    • \IK16 Inserts a hyper-link reference to the int16 integer kind used in the library.
    • \IK32 Inserts a hyper-link reference to the int32 integer kind used in the library.
    • \IK64 Inserts a hyper-link reference to the int64 integer kind used in the library.

    For an up-to-date list of all available aliases, check the value of the Doxygen ALIASES option in config.txt.

  • Avoid the insertion of an empty documentation line between any two lines of a single Doxygen paragraph. This is crucial when the whole paragraph is indented by a vertical line as is done by Doxygen for \warning, \remark, \note and other similar tags. Example: See the aforementioned tags in the example below.
  • Example usage ⛓
    The following is an example documentation for a procedure: 
          !>  \brief
      !>  Generate and return the variance of the input array of shape `(np)` or `(nd,np)` or `(np,nd)` where `nd` is the number of
      !>  data dimensions (the number of data attributes) and `np` is the number of data points.
      !>
      !>  \param[in]  Sample  :   The input `contiguous` array of type `real` of kind \RKALL of shape `(np)`, `(nd,np)`, or `(np,nd)`
      !>                          containing the sample. If `Sample` is a 2D array, then the direction along which the variance is computed
      !>                          is dictated by the optional input argument `dim`.
      !>  \param[in]  Weight  :   The contiguous vector of shape `(np)` either type `real` of the same kind as the input `Sample` or type `integer`
      !>                          of kind \IKALL, containing the corresponding weight of each data points in `Sample`
      !>                          (**optional**, default = a vector of ones).
      !>  \param[in]  Mean    :   The input scalar or contiguous vector of shape `(nd)` of the same type and kind as the input `Sample` containing
      !>                          the `Sample` mean along the (optionally) specified dimension `dim`. If the input `Sample` is a 1D array, then `Mean`
      !>                          must be a scalar. Otherwise, if `Mean` is a 2D array, then `Mean` must be a vector whose size is the same as
      !>                          the size of at least one of the dimensions of `Sample`.
      !>                          (**optional**. If missing, then the input argument `shifted` must be present indicating whether the input `Sample`
      !>                          is already centered at the origin or it has to be shifted to the origin by the procedure).
      !>  \param[in]  shifted :   The input `logical` of default kind \LK indicating whether the input `Sample` is already centered at the origin or
      !>                          the it has to be shifted to the origin by the procedure (**optional**. If missing, then the input
      !>                          argument `Mean` must be present).
      !>  \param[in]  biased  :   The input `logical` of default kind \LK indicating whether the output variance should be corrected for small sample-size
      !>                          bias. Set this argument to `.false.` to avoid biased variance computation, in particular, when the sample size `np`
      !>                          is small.
      !>  \param[in]  dim     :   An integer of default kind \IK indicating which dimension of the input `Sample` iterates over the individual data points.
      !>                          If `dim = 1` or `dim /= 2`, the input `Sample` is assumed to have the shape `(np,nd)`.
      !>                          If `dim = 2`, the input `Sample` is assumed to have the shape `(nd,np)`
      !>                          (**optional**, default = `2`. **This input argument is available only if the input `Sample` is a 2D array**.).
      !>
      !>  \return
      !>  `variance`          :   The output variance of the input sample of the same type and kind as the input `Sample`.
      !>                          It is a scalar only if the input `Sample` is a 1D array. Otherwise, it is an `allocatable` array of shape `(nd)`.
      !>
      !>  \warnpure
      !>
      !>  \note
      !>  One also use the concise Fortran syntax to achieve the same goal as this function:
      !>  \code{fortran}
      !>
      !>       mean = sum(Weight*Sample) / sum(Weight)
      !>       variance = sum( (Weight*(Sample-mean))**2 ) / (sum(Weight)-1)
      !>
      !>  \endcode
      !>  But the above concise version will be slightly slower as it involves three loops instead of two.
      !>
      !>  \see
      !>  [genMean()](@ref SampleMean_mod::genMean) <br>
      !>
      !>  \example
      !>  \include{lineno} example/kernel/Sample_mod/genVariance/main.f90
      !>  \compile
      !>  \output
      !>  \include{lineno} example/kernel/Sample_mod/genVariance/main.out
      !>
      !>  \test
      !>  [test_genVariance](@ref Test_Sample_mod::test_genVariance)
      !>
      !>  \todo
      !>  The performance of this code can improved.
      !>
      !>  \bug
      !>  This code used to have a well-known bug in version 1.1, but is now resolved.
      !>
      !>  \final
      !>
      !>  \author
      !>  Amir Shahmoradi, Monday 02:15 AM, September 27, 2021, Dallas, TX

    The above example documentation snippet will generate an HTML similar to this documentation. Note the lack of an empty line among the commands that immediately follow \example. This is essential to keep the entire example section in the same paragraph.