mv_jobflow.sas File Reference

Execute a series of job flows. More...

Go to the source code of this file.

Detailed Description

Very (very) simple flow manager. Jobs execute in sequential waves, all previous waves must finish successfully.

The input table is formed as per below. Each observation represents one job. Each variable is converted into a macro variable with the same name.

Input table (minimum variables needed)

  • _PROGRAM - Provides the path to the job itself
  • FLOW_ID - Numeric value, provides sequential ordering capability. Is optional, will default to 0 if not provided.
  • _CONTEXTNAME - Dictates which context should be used to run the job. If blank, or not provided, will default to SAS Job Execution compute context.

Any additional variables provided in this table are converted into macro variables and passed into the relevant job.

_PROGRAM FLOW_ID (optional) _CONTEXTNAME (optional)
/Public/jobs/somejob1 0 SAS Job Execution compute context
/Public/jobs/somejob2 0 SAS Job Execution compute context

Output table (minimum variables produced)

  • _PROGRAM - the SAS Drive path of the job
  • URI - the URI of the executed job
  • STATE - the completed state of the job
  • TIMESTAMP - the datetime that the job completed
  • JOBPARAMS - the parameters that were passed to the job
  • FLOW_ID - the id of the flow in which the job was executed

https://i.imgur.com/nZE9PvT.png

To avoid hammering the box with many hits in rapid succession, a one second pause is made between every request.

Example

First, compile the macros:

filename mc url
"https://raw.githubusercontent.com/sasjs/core/main/all.sas";
%inc mc;

Next, create some jobs (in this case, as web services):

filename ft15f001 temp;
parmcards4;
  %put this is job: &_program;
  %put this was run in flow &flow_id;
  data ;
    rand=ranuni(0)*&macrovar1;
    do x=1 to rand;
      y=rand*&macrovar2;
      if y=100 then abort;
      output;
    end;
  run;
;;;;
%mv_createwebservice(path=/Public/temp,name=demo1)
%mv_createwebservice(path=/Public/temp,name=demo2)

Prepare an input table with 60 executions:

data work.inputjobs;
  _contextName='SAS Job Execution compute context';
  do flow_id=1 to 3;
    do i=1 to 20;
      _program='/Public/temp/demo1';
      macrovar1=10*i;
      macrovar2=4*i;
      output;
      i+1;
      _program='/Public/temp/demo2';
      macrovar1=40*i;
      macrovar2=44*i;
      output;
    end;
  end;
run;

Trigger the flow

%mv_jobflow(inds=work.inputjobs
  ,maxconcurrency=4
  ,outds=work.results
  ,outref=myjoblog
)

data _null_;
  infile myjoblog;
  input; put _infile_;
run;
Parameters
[in]access_token_var=The global macro variable to contain the access token
[in]grant_type=valid values:
  • password
  • authorization_code
  • detect - will check if access_token exists, if not will use sas_services if a SASStudioV session else authorization_code. Default option.
  • sas_services - will use oauth_bearer=sas_services
[in]inds=The input dataset containing a list of jobs and parameters
[in]maxconcurrency=The max number of parallel jobs to run. Default=8.
[in]raise_err=0Set to 1 to raise SYSCC when a job does not complete succcessfully
[in]mdebug=set to 1 to enable DEBUG messages
[out]outds=The output dataset containing the results
[out]outref=The output fileref to which to append the log file(s).
Version
VIYA V.03.05
Author
Allan Bowe, source: https://github.com/sasjs/core

SAS Macros

Definition in file mv_jobflow.sas.