How to perform gradient checking in a neural network with batch normalization?

I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with gradient checking.

A few days ago, I implemented batch normalization using the formulas provided by the original paper. However, in contrast with learning/momentum/regularization, the batch normalization procedure behaves differently during fit and predict phases – both needed for gradient checking. As we fit the network, batch normalization computes each batch mean and estimates the population’s mean to be used when we want to predict something.

In a similar way, I know we may not perform gradient checking in a neural network with dropout, since dropout turns some gradients to zero during fit and is not applied during prediction.

Can we perform gradient checking in NN with batch normalization? If so, how?

All topic

Batch conversion of arbitrary formats to djvu, mobi, epub – not with calibre

I have a few thousand files, mostly PDF, many postscript, that I need to convert to several other formats. I can’t use calibre (don’t get me started) – I have considered writing scripts to dump pages to image files and using dubious cli programs to compile the images to these formats, but thought I would ask here before reinventing the wheel. Hopefully someone has an elegant solution. Epub is just zipped html with a specific file layout, the others ( djvu, mobi ) seem a bit different.

All topic

multiple variables inside a single variable batch

I’m having trouble with putting variables inside of variables in Batch, I have a similar section of code in another part of my file that works fine, it just doesn’t use the %reverseAdjustment% variable, so I believe that is where the issue lies. The issue in question is the variable %input% is outputting ‘2rotor326’ or ‘echo is off’ instead of the value held by the variable %2rotor326% (which is 15).

set rotor3=3
set /a orientationOffset3=0
set 2rotor326=15
set rotor315=26

these variables are set as examples, but if it works with these values and the output is 15 it is working correctly.

set /a rotorSector=%input%+!orientationOffset%rotor3%!
if %rotorSector% gtr 26 set /a rotorSector=%rotorSector%-26
set reverseAdjustment=!rotor%rotor3%%rotorSector%!
set input=!2rotor%rotor3%%reverseAdjustment%!
echo %input%
pause

Some places I have looked have mentioned the way that batch deals with setting variables and not executing them in the same block, however, I did try putting each section into different ‘headings’ (sorry I don’t know the correct term) and it still didn’t work. And yes I have used ‘enabledelayedexpansion’

All topic

ACDSee alternatives for batch editing images

I am looking for free, preferably open, alternatives to ACDSee for batch editing work. While I can do much of the work well on ACDSee, it’s not entirely satisfactory despite having to pay for it. I need at least the following batch editing functions:

  • Resize using either height or width while maintaining aspect ratio
  • Auto contrast
  • text overlays
  • and occasionally, cropping
  • oh, I make extensive use of renaming features as well

Couple of issues with ACDSee are: I always need to highlight the Exposure section or auto contrast will not be done despite it being saved in the preset; and I can’t define, move around the cropping box, forcing me to manually crop tons of images.

I’m not an advanced, or “power photo-editor”. I only require the basic stuff I described to be automated. My personal feature wish list (I’m pretty sure something so niche doesn’t exist) would be text overlay based on the image names (images are named as image-1_1, image-1_2 or image-2_c1_1, image-2_c1_2, and text overlay would Image-1 and Image-2 C1 and Image-2 C2).

I tried digiKam, but damn that thing is huge. It runs very slowly on my Pentium 4 and 1.5 GB RAM. On top of being a program with over 1 GB of files, the KDE library it uses is always slow regardless of it running on either Windows or Linux.

All topic

Batch File FOR Loop 7-zip extract specific file extension

I am trying to create a batch file to recursively extract multiple archives in multiple sub-directories from a source directory to an output directory while keeping archive folder structure. The below example works perfectly EXCEPT that I need to extract only one specific file type. What do I need to change in order to make this work?

@ECHO ON

SET source=C:UsersuserDesktoptest
FOR /F "TOKENS=*" %%F IN ('DIR /S /B "%source%*.zip"') DO "C:Program Files7-Zip7z.exe" x "%%~fF" -o"C:UsersuserDesktopWorkingCustom Textures*"
FOR /F "TOKENS=*" %%F IN ('DIR /S /B "%source%*.7z"') DO "C:Program Files7-Zip7z.exe" x "%%~fF" -o"C:UsersuserDesktopWorkingCustom Textures*"
FOR /F "TOKENS=*" %%F IN ('DIR /S /B "%source%*.rar"') DO "C:Program Files7-Zip7z.exe" x "%%~fF" -o"C:UsersuserDesktopWorkingCustom Textures*"
PAUSE

I sincerely appreciate the help in advance, and any thorough resources on batch scripting that I could be directed to for future references would be greatly appreciated as well, as I am trying to learn.

All topic

Simple CRUD application which also executes a batch file

New to application development – predominantly working on back-end.

I need to develop a very simple web app which has a couple of fields for start and begin months. once entered, it needs to update a table in SQL Server database. And on click of a button, execute a batch file which resides on a server.

What is the easiest way to go about this with the least learning curve. I do some backend scripting with Python so Flask maybe an easier curve. But would it be a good candidate for something like this?

I looked a bit online and there are quite a few with many moving parts which I am not sure I need to invest in – so hoping for any suggestions on something easy and simple.

All topic

how to send 200 http request in one minute in Batch class in salesforce [on hold]

how to send 200 http request in one minute in Batch class in salesforce

All topic

Usar Batch para fazer FTP e baixar sempre o arquivo mais atual

Tenho um script em .BAT que faz um FTP em servidor para baixar um arquivo, porém na pasta existe varios arquivos (mget REC_PENDaaaammddhhmm.csv) de horas anteriores, gostaria de copiar sempre o arquivo mais atual, é possível ?

All topic

Rename files sequentially from input using batch file?

The company I work for processes bills, and I regularly have to check these bills for accuracy by comparing what was entered into the database vs. the actual bill image. Any time I find an error in processing, I must delete the bill from the database and send the image to a processor to re-enter the bill correctly. However, our system frequently encounters a glitch where if I try to upload the bill to the server that gets it to the processor, it says that a duplicate file exists. To bypass this issue I have to manually rename each image with a new name (we use rolls of control numbers to name our images).

I’d like to automate this process so that I can rename these files quickly in sequential order. I would like my batch file to have an input so that I can enter in the starting control number and it will rename each file in the directory in sequential order, starting with the control number I enter +1 to each additional file in the directory, for example it would go like this:

Batch file asks me starting control #. I enter in 6654821.

  • bill3210851.pdf renamed to 6654821
  • imagebill654.pdf renamed to 6654822
  • random_name.pdf renamed to 6654823
  • billerror.pdf renamed to 6654824
  • sendtomanual.pdf renamed to 6654825

The original name of the file is not at all relevant, they don’t need to remain in any specific order as long as they are renamed sequentially.

So far what I have is this:

@echo OFF
title Batch Rename
color 5f

echo Hello! I'm the batch pdf renamer. Please enter the starting control number:
set /p start=

ren %userprofile%desktopneedrenamed*.pdf "%p%.pdf"

pause

Obviously this doesn’t address renaming the files sequentially. I’m kind of stuck here.

All topic

Script-thrown exception in Test class for Batch apex callout

I have a batch class which makes multiple http callouts to an HR system and then with the absences returned checks for existing absences and upserts. However when I try and run test class for the batch class, I get the error “Script-thrown exception”

My batch class (please note I have not included the wrapper calsses CiphrWrapper, Ciphr and CiphrAbsenceDetail)

global class CiphrBatchJob implements Database.Batchable, Database.AllowsCallouts{
String query;
//final ID UKLog = Schema.getGlobalDescribe().get('Integration_Log__c').getDescribe().getRecordTypeInfosByName().get('UK Log').getRecordTypeId();
String Endpoint = 'https://****/***/absence?EmployeeNumber=[in]';
String DetailEndpoint;
private String API_Key;
private date cutOffDate= Date.newInstance(2010,01,01); //Change date to current year

global Database.QueryLocator start(Database.BatchableContext bc){    
    query = 'Select KC_Employee_Reference__c from KimbleOne__Resource__c where KC_Employee_Reference__c <> null';
    return Database.getQueryLocator(query);
}

global void execute(Database.BatchableContext BC, List scope){
    for(Integer i=0;i LogList = new List();
    Integration_log__c log = new Integration_log__c();
    log.recordTypeID = UKLog;
    log.Job_Name__c = 'Ciphr Get Absence Request';
    log.Request_Body__c = 'Get '+Endpoint;*/

    try{
        HttpResponse response = http.send(request);
        result = CiphrWrapper.parse(response.getBody());
        /*if(response.getBody().length() >= 131001){
           log.Response_Body__c = String.valueOf(response.getBody()).subString(0,131000); 
        }else{
           log.Response_Body__c = response.getBody(); 
        }
        log.Status__c = response.getStatus();
        log.Status_Code__c = response.getStatusCode();*/     
    }catch(Exception e){
        System.debug('Exception during call--> '+e.getMessage());
        //log.Exception__c = e.getMessage();
        sendEmail('Call to Ciphr failed','Exception Occurred',e.getMessage(),true,Endpoint);
    }
    //LogList.add(log);

    Map CRMap =  new Map();
    Map CADMap = new Map();
    Map CAMap = new Map();
    DateTime dt; Date d;

    if(result <> null){
        //Populate Resources
        for(CiphrWrapper.Absence ab: result.Absence){
            dt = DateTime.ValueOf(ab.Start.replace('T',' '));
            d = Date.newinstance(dT.year(), dT.month(), dT.day());
            if(d >= /*System.today()*/ cutOffDate){     //<---- Change Date here
                Ciphr.Resource CR = new Ciphr.Resource();
                CR.employeeNumber = ab.EmployeeNumber;
                CR.absences = new List();
                CRMap.put(CR.employeeNumber, CR);
            }
        }       

        //Populate Absences in Resources
        for(CiphrWrapper.Absence ab: result.Absence){
            dt = DateTime.ValueOf(ab.Start.replace('T',' '));
            d = Date.newinstance(dT.year(), dT.month(), dT.day());
            if(CRMap.containsKey(ab.employeeNumber) && d >= /*System.today()*/ cutOffDate){    //<-- Change date here
                Ciphr.Absence  CA = new Ciphr.Absence();
                CA.AbsenceID = Integer.valueOf(ab.ID);
                CA.employeeNumber = ab.EmployeeNumber;
                CA.absenceDetails = new List();
                CAMap.put(CA.AbsenceID, CA);
                CRMap.get(CA.employeeNumber).absences.add(CA);
            }
        }
        System.debug('CRMap: '+CRMap);            

        //Calling Absences
        for(Ciphr.Resource cr: CRMap.values()){
            DetailEndpoint = 'https://****.***/***/absenceDetail?AbsenceID=[in]';
            Integer i = 0; 
            for(Ciphr.Absence ca: cr.absences){
                if(i==cr.absences.size()-1){
                    DetailEndpoint += String.valueOf(ca.AbsenceID);
                }else{
                    DetailEndpoint += String.valueOf(ca.AbsenceID)+',';
                }
                i++;
            }
            System.debug('Details Endpoint for emp: '+cr.employeeNumber+' is:'+DetailEndpoint);
            HttpRequest requestAbsenceDetail = new HttpRequest();

            requestAbsenceDetail.setEndpoint(DetailEndpoint);
            requestAbsenceDetail.setMethod('GET');
            requestAbsenceDetail.setHeader('Accept', 'application/json');
            requestAbsenceDetail.setHeader('Authorization', 'apikey '+API_Key);

            CiphrAbsenceDetail resultAbsenceDetails;
            /*Integration_Log__c DetailLog = new Integration_log__c();
            DetailLog.RecordTypeId = UKLog;
            DetailLog.Job_Name__c = 'Ciphr Get Absence Detail Request';
            DetailLog.Request_Body__c = 'Get '+DetailEndpoint;*/

            try{
                HttpResponse responseDetails = http.send(requestAbsenceDetail);
                resultAbsenceDetails = CiphrAbsenceDetail.parse(responseDetails.getBody());
                /*DetailLog.Response_Body__c = responseDetails.getBody();
                DetailLog.Status__c = responseDetails.getStatus();
                DetailLog.Status_Code__c = responseDetails.getStatusCode();*/
            }catch(Exception e){
                System.debug('Exception during Absence detail call for EmployeeNumber --> '+cr.employeeNumber+' '+e.getMessage());
                //DetailLog.Exception__c = e.getMessage();
                sendEmail('SF Ciphr Detail Call failed','Exception Occurred in Detail Call',e.getMessage(),true,DetailEndpoint);
            }//LogList.add(DetailLog);                
            if(resultAbsenceDetails<> null){
                for(Ciphr.Absence ca: cr.absences){
                    for(CiphrAbsenceDetail.AbsenceDetail cad: resultAbsenceDetails.AbsenceDetail){
                        if(ca.AbsenceID == cad.AbsenceID){
                            Ciphr.AbsenceDetail cad1 = new Ciphr.AbsenceDetail();
                            cad1.employeeNumber = ca.employeeNumber;
                            cad1.AbsenceID = cad.AbsenceID;
                            cad1.End_Z = cad.End_Z;
                            cad1.Start = cad.Start;
                            cad1.ID = Integer.valueOf(cad.AbsenceID)+'_'+Integer.valueOf(cad.ID);
                            cad1.Hours = cad.Hours;
                            CADMap.put(cad1.ID, cad1);
                            CAMap.get(ca.AbsenceID).absenceDetails.add(cad1);
                            ca.AbsenceDetails.add(cad1);
                        }
                    }

                }    
            }
            System.debug('CAMap: '+CAMap);     
        }        
        System.debug('CRMap: '+CRMap);    
    } 
    //Insert Logs         //Insert LogList;
    //Retrieve current absences greater than the cutoffdate from Kimble
    List kimbleCurrentAbsences = [SELECT KimbleOne__ExternalId__c, KimbleOne__Resource__r.KC_Employee_Reference__c, KimbleOne__EntryUnits__c, KimbleOne__EntryDate__c FROM KimbleOne__TimeEntryImportLine__c where KimbleOne__EntryDate__c >=: cutOffDate And KimbleOne__Resource__r.KC_Employee_Reference__c in:CRMap.keyset()];
    Map KimbleCurrentAbsencesMap = new Map();
    List> PayLoad = new List>();

    if(KimbleCurrentAbsences.size()>0){
        for(KimbleOne__TimeEntryImportLine__c kab: kimbleCurrentAbsences){
            KimbleCurrentAbsencesMap.put(kab.KimbleOne__ExternalId__c, kab);
        }
    }

    //Check and remove absence details if they exist in Kimble with same absence detail Id and same no. of hours 
    for(Ciphr.AbsenceDetail cad: CADMap.Values()){
        if(KimbleCurrentAbsencesMap.containsKey(cad.id)){
            if(cad.hours == KimbleCurrentAbsencesMap.get(cad.id).KimbleOne__EntryUnits__c)
            {
                CADMap.remove(cad.id);
                KimbleCurrentAbsencesMap.remove(cad.id);
            }
        }
    }

    //Check if there are any absence details in Kimble but are not Ciphr i.e. absence details that were deleted in Ciphr so add a 0 against the absence deleted in Ciphr
    for(KimbleOne__TimeEntryImportLine__c kab: KimbleCurrentAbsencesMap.values()){
        if(!CadMap.containsKey(kab.KimbleOne__ExternalId__c)){
            Ciphr.AbsenceDetail cad = new Ciphr.AbsenceDetail();
            cad.employeeNumber = KimbleCurrentAbsencesMap.get(kab.KimbleOne__ExternalId__c).KimbleOne__Resource__r.KC_Employee_Reference__c;
            cad.Start = KimbleCurrentAbsencesMap.get(kab.KimbleOne__ExternalId__c).KimbleOne__EntryDate__c.format();
            cad.ID = kab.KimbleOne__ExternalId__c;
            cad.Hours = 0;
            CADMap.put(kab.KimbleOne__ExternalId__c,cad);
        }
    }


    //Prepare the payload
    for(Ciphr.AbsenceDetail cad: CADMap.Values()){
        Map MaptoSerialize = new Map{'resource' => cad.employeeNumber,'activity' => 'a5z1w0000004CH5AAM', 'date' => Date.valueOf(cad.start), 'hours' => cad.Hours, 'CIPHR_Id' => cad.ID};
        PayLoad.add(MaptoSerialize);
    }    

    String responseOfPayloadPush = SendPayload(PayLoad);

    System.debug('*******************Payload******************n'+json.serializePretty(PayLoad));
    System.debug('Response of the payload push: '+responseOfPayloadPush); //Uncomment for testing

    //Below only for testing
    Attachment att = new Attachment();
    att.parentID = '001w000001W37mWAAR'; //Attaches to account "jon_singer_test_UK_SOHO" 
    att.ContentType = 'text/plain';
    att.name = 'Payload '+Date.today()+'.txt';
    if(String.valueOf(json.serializePretty(PayLoad)).length()>131000){
        att.body = Blob.valueOf(String.valueOf(json.serializePretty(PayLoad)).substring(0,131000));
    }else{
        att.body = Blob.valueOf(String.valueOf(json.serializePretty(PayLoad)));
    }
    insert att;

    Attachment att1 = new Attachment();
    att1.parentID = '001w000001W37mWAAR'; //Attaches to account "jon_singer_test_UK_SOHO" 
    att1.ContentType = 'text/plain';
    att1.name = 'Payload Push Response'+Date.today()+'.txt';
    att1.body = Blob.valueOf(responseOfPayloadPush);
    insert att1;        
    //Above for Testing only
}

global void finish(Database.BatchableContext BC){
    sendEmail('SF Ciphr Records Processed','SF Ciphr Batch Process Completed','SF Ciphr Batch Process Completed.',false,'');
}

private String getApiKey(){
    Ciphr_Kimble__c ck = Ciphr_Kimble__c.getInstance();
    String key = ck.EncryptionKey__c;
    String APIKey = ck.EncryptedCiphrKey__c;
    Blob decrypted = Crypto.decryptWithManagedIV('AES128', EncodingUtil.base64Decode(key), EncodingUtil.base64Decode(APIKey));
    APIKey = decrypted.toString();
    //System.debug('API Key getApiKey '+APIKey);
    return APIKey; 
}

private void sendEmail(string displayName, string subject, String messageBody, boolean exceptionOccurred, String endpoint){
    Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
    mail.setToAddresses(new String[] {'****@***.***'});
    //mail.setToAddresses(new String[] {'*****@****.***'});
    mail.setReplyTo('batch@acme.com');
    mail.setSenderDisplayName(displayName);
    mail.setSubject(subject);
    if(exceptionOccurred){
        mail.setPlainTextBody('Exception: '+messageBody+'nnFor URL: '+endpoint);
    }else{
        mail.setPlainTextBody(messageBody);
    } 
    Messaging.sendEmail(new Messaging.SingleEmailMessage[] { mail });
}

private String SendPayload(List> payload){
    String sfdcURL = URL.getSalesforceBaseUrl().toExternalForm(); 
    String restAPIURL = sfdcURL + '/services/apexrest/***/**/****/***';  

    HttpRequest httpRequest = new HttpRequest();
    httpRequest.setEndpoint(restAPIURL); 
    httpRequest.setMethod('POST'); 
    httprequest.setHeader('Content-Type', 'application/json');
    httprequest.setHeader('Accept','application/json');
    httpRequest.setHeader('Authorization', 'OAuth ' + UserInfo.getSessionId());        
    httpRequest.setHeader('Authorization', 'Bearer ' + UserInfo.getSessionID());  
    httpRequest.setBody(json.serializePretty(PayLoad));
    String response = '';
    try {  
        Http http = new Http();   
        HttpResponse httpResponse = http.send(httpRequest);  
        System.debug('>> Response of payload >> '+httpResponse.getStatusCode());
        response = 'Status Code: '+httpResponse.getStatusCode()+' Response: '+httpResponse.getBody();
    } catch(Exception e) {  
        System.debug('ERROR: '+ e.getMessage());  
        response = e.getMessage();
        sendEmail('Payload Sending to Kimble failed','Exception Occurred Payload Push',e.getMessage(),true,restAPIURL);
    }  
    System.debug(' ** response ** : ' + response );
    return response;
}}

My Test class

@isTest(SeeAllData = True) //In progress
public class CiphrBatchJobTest {
static testmethod void doTest(){
    Test.startTest();    
    CiphrBatchJob c = new CiphrBatchJob();    
    Test.setMock(HttpCalloutMock.class, new MockHttpResponseGenerator_Ciphr());        
    ID BatchProcessID = Database.executeBatch(c,50);
    Test.stopTest();
}
}

My http mock class:

@isTest
global class MockHttpResponseGenerator_Ciphr implements HttpCalloutMock{
 // Implement this interface method
global HTTPResponse respond(HTTPRequest req) {               
    // Create a fake response
    if(req.getBody().contains('EmployeeNumber')){
        HttpResponse res = new HttpResponse();
        res.setHeader('Content-Type', 'application/json');
        res.setBody('{"Absence": [{"ID": "1463","Start": "1995-08-08T00:00:00","End": "1995-08-20T00:00:00","EmployeeNumber": "10187","Forenames": "Jonathan","Surname": "Christopher"}]}');
        res.setStatusCode(200);
        return res;    
    }else if(req.getBody().contains('AbsenceID')){
        HttpResponse res = new HttpResponse();
        res.setHeader('Content-Type', 'application/json');
        res.setBody('{"AbsenceDetail": [{"AbsenceID": "1463","Start": "1995-08-08T00:00:00","End": "1995-08-20T00:00:00","ID": 8763 ,"Hours": 7.40}]}');
        res.setStatusCode(200);
        return res;    
    }else{
        return null;
    }

}
}

I know that seeAllData=true is a bad practice but I am not able to insert into the KimbleOne__Resource__c object and subsequently KimbleOne__TimeEntryImportLine__c as these are 3rd party objects and we were advised not to insert as there are 30+ objects that would need inserted etc (basically it’s too complex)

Are you able to help please, as I’m unsure on how to go about it?

All topic