How to improve testing accuracy when training accuracy is high?

Following-up my question about my over-fitting network

My deep neural network is over-fitting :
enter image description here

I have tried several things :

  • Simplify the architecture
  • Apply more (and more !) Dropout
  • Data augmentation

But I always reach similar results : training accuracy is eventually going up, while validation accuracy never exceed ~70%.

I think I simplified enough the architecture / applied enough dropout, because my network is even too dumb to learn anything and return random results (3-classes classifier => 33% is random accuracy), even on training dataset :
enter image description here

My question is : This accuracy of 70% is the best my model can reach ?

If yes :

  • Why the training accuracy reach such high scores, and why so fast, knowing this architecture seems to be not compatible ?
  • My only option to improve the accuracy is then to change my model, right ?

If no :

  • What are my options to improve this accuracy ?

I’v tried a bunch of hyperparameters, and a lot of time, depending of these parameters, the accuracy does not change a lot, always reaching ~70%. However I can’t exceed this limit, even though it seems easy to my network to reach it (short convergence time)

Edit

Here is the Confusion matrix :

enter image description here

I don’t think the data or the balance of the class is the problem here, because I used a well-known / explored dataset : SNLI Dataset

And here is the learning curve :

enter image description here

Note : I used accuracy instead of error rate as pointed by the resource of Martin Thoma

It’s really ugly one. I guess there is some problem here.
Maybe the problem is that I used the result after 25 epoch for every values. So with little data, training accuracy don’t really have time to converge to 100% accuracy. And for bigger training data, as pointed in earlier graphs, the model overfit so the accuracy is not the best one.

Testing that training and validation data sets (not vectors) have been drawn from the same population in R

I have two data frames in R (test + validation) and understand that predictions for the validation dataset provided are only possible if we can confirm that training and validation datasets have been drawn from the same population.

In this [Non-parametric test if two samples are drawn from the same distribution post it has been discussed to use the Kolmogorov–Smirnov test. However, in R the ks.test seems to only be possible to conduct on vectors. Would I therefore have to do this test for each numeric variable independently or is there a test/way to do that on the whole data set at once?

Thanks 🙂

Approaches for testing huge data files in text format

My current assignment requires me to test data in a large number of text files ~ 200, usually ranging between 1GB to 20GB.
The data is in same schema with coordinates.

As of now the approach is to load these files in a database (PostgreSQL) and then fire test cases using Python.
Running test on all the files (~80 GB) takes anywhere around 7/8 hours.

Even though the process is majorly automated, there are few drawbacks, namely a lack of proper reporting and manual intervention.

I want to optimise and improve this process.

Is the above approach is the best way of testing data?
Are there better ways to do this?

What are the approaches you guys use to test data?

Can data languages like R / Scala be useful for this kind of testing?

Looking for suggestions and pointers.

Thanks.

Custom question lists is available for testing

We are excited to release the first phase of custom question lists for alpha testing. Check out Custom question lists: finding questions you can answer for a refresher on the feature. A reminder that the functionality being released is a new experience for selecting the question lists and new filter/sort controls. The ability to save and reuse “custom lists” comes in phase II.

Note: This functionality is being released on Stack Overflow, Meta Stack Overflow and Meta Stack Exchange. Once it is out of alpha, it will be available on all network sites.

How to opt into the alpha

1. Go to your user preference

2. Check “Opt me in” for the “Custom Question List” feature

Custom Question List alpha preference
note: set it once and the feature (when active) will be available on SO, MSO, MSE.

3. Wait.

You won’t be immediately opted in. At first your status will be “Pending”. We are limiting the number of people who have access. This will let us find/fix issues with less disruption. And, it will let us manage the load that this new feature puts on the servers (this could be more of an issue in phase II).

4. Test away.

When you do get access to the feature, please try it out. Log bugs as answers on this post.

Schedule for future improvements

This feature will roll out in four phases:

  • Phase I: Filter and sort controls to let you modify a question list view (Julyish)
  • Phase II: Saved custom lists (In progress, ETA November)
  • Phase III: Notifications on custom lists (December)
  • Phase IV: Final fixes and official release (January)

Testing whether an integer is the sum of two squares

Is there a fast (probabilistic or deterministic) algorithm for determining whether an integer $n$ is a sum of two squares?

By “fast” here I mean polynomial time (i.e. time $O((log n)^{O(1)})$). Note that I am interested only in whether the integer can be represented in such a way, not in how it is represented.

Since a fast algorithm is required, it will not do to use factorization.

It would be odd if this turned out to be harder than detecting primality, since prime numbers are rarer.

(This is a question that came up in a talk I just gave.)

Release tokens function on the ERC20 is not working during ICO testing with remix!

So I modified the ERC20 token a little bit added two extra functions so that my ICO token contract can call it to send some tokens to the people.

This is my erc20 token code.

pragma solidity ^0.4.24;

contract SafeMath {

    function safeAdd(uint a, uint b) public pure returns (uint c) {
        c = a + b;
        require(c >= a);
    }

    function safeSub(uint a, uint b) public pure returns (uint c) {
        require(b <= a);
        c = a - b;
    }

    function safeMul(uint a, uint b) public pure returns (uint c) {
        c = a * b;
        require(a == 0 || c / a == b);
    }

    function safeDiv(uint a, uint b) public pure returns (uint c) {
        require(b > 0);
        c = a / b;
    }
}


contract ERC20Token is SafeMath {

    mapping(address => uint256) balances;
    mapping(address => mapping(address => uint)) allowed;
    uint _totalSupply;

    function totalSupply() public constant returns (uint) {
        return _totalSupply  - balances[address(0)];
    }

    function balanceOf(address tokenOwner) public constant returns (uint balance) {
        return balances[tokenOwner];
    }

    function transfer(address to, uint tokens) public returns (bool success) {
        balances[msg.sender] = safeSub(balances[msg.sender], tokens);
        balances[to] = safeAdd(balances[to], tokens);
        emit Transfer(msg.sender, to, tokens);
        return true;
    }

    function approve(address spender, uint tokens) public returns (bool success) {
        allowed[msg.sender][spender] = tokens;
        emit Approval(msg.sender, spender, tokens);
        return true;
    }

    function transferFrom(address from, address to, uint tokens) public returns (bool success) {
        balances[from] = safeSub(balances[from], tokens);
        allowed[from][msg.sender] = safeSub(allowed[from][msg.sender], tokens);
        balances[to] = safeAdd(balances[to], tokens);
        emit Transfer(from, to, tokens);
        return true;
    }

    function allowance(address tokenOwner, address spender) public constant returns (uint remaining) {
        return allowed[tokenOwner][spender];
    }

    event Transfer(address indexed from, address indexed to, uint tokens);
    event Approval(address indexed tokenOwner, address indexed spender, uint tokens);

}

contract Owned {

    address public owner;
    address public newOwner;

    event OwnershipTransferred(address indexed _from, address indexed _to);

    constructor() public {
        owner = msg.sender;
    }

    modifier onlyOwner {
        require(msg.sender == owner);
        _;
    }

    function transferOwnership(address _newOwner) public onlyOwner {
        newOwner = _newOwner;
    }

    function acceptOwnership() public {
        require(msg.sender == newOwner);
        emit OwnershipTransferred(owner, newOwner);
        owner = newOwner;
        newOwner = address(0);
    }
}

contract OurToken is ERC20Token, Owned {

    string public symbol;
    string public  name;
    uint public decimals;
    address public crowdsaleAddress;
    uint public ICOEndTime = 0;

   modifier onlyCrowdsale {
      require(msg.sender == crowdsaleAddress);
      _;
   }

   modifier afterCrowdsale {
      require(now > ICOEndTime);
      _;
   }

  constructor (string _symbol,string _name,uint _decimals,uint totalSupply) public {

      symbol = _symbol;
      name = _name;
      decimals = _decimals;
      _totalSupply = totalSupply * 10 ** decimals;
      owner = msg.sender;
      balances[msg.sender] = _totalSupply;
   }

    function transfer(address to, uint tokens) public afterCrowdsale returns (bool) {
        super.transfer(to,tokens);
    }

    function transferFrom(address from, address to, uint tokens) public afterCrowdsale returns (bool) {
        super.transferFrom(from,to,tokens);
    }

    function emergencyExtract() external onlyOwner {
        owner.transfer(address(this).balance);
    }

   function setICOEndtime(uint time) public onlyOwner {
        require(time>0);
        ICOEndTime = time;
   }

   function setCrowdsale(address _crowdsaleAddress) public onlyOwner {
      require(_crowdsaleAddress != address(0));
      crowdsaleAddress = _crowdsaleAddress;
   }

   function releaseTokens(address _receiver, uint256 _amount) public onlyCrowdsale {
      require(_receiver != address(0));
      require(_amount > 0);
      balances[owner] = safeSub(balances[owner], _amount);
      balances[_receiver] = safeAdd(balances[_receiver], _amount);
      emit Transfer(owner, _receiver, _amount);

   }

   function refundTokens(address _sender, uint256 _amount) public onlyCrowdsale {
      balances[_sender] = safeSub(balances[_sender], _amount);
      balances[owner] = safeAdd(balances[owner], _amount);
      emit Transfer(_sender,owner,_amount);

   }
}

ICO contract

pragma solidity ^0.4.24;


contract OurToken {

   function balanceOf(address tokenOwner) public constant returns (uint balance);

   // To release tokens to the address that have send ether.
   function releaseTokens(address _receiver, uint _amount) public;

   // To take back tokens after refunding ether.
   function refundTokens(address _receiver, uint _amount) public;

}

contract ICO {

   uint public icoStart;
   uint public icoEnd;
   uint public tokenRate;
   OurToken public token;   
   uint public fundingGoal;
   uint public tokensRaised;
   uint public etherRaised;
   address public owner;
   address public extractor;

   modifier whenIcoCompleted {
      require(now>icoEnd);
      _;
   }

   modifier onlyOwner {
      require(msg.sender == owner);
      _;
   }

   modifier onlyExtractor {
      require(msg.sender == extractor || msg.sender == owner);
      _;
   }

   constructor(uint256 _icoStart, uint _icoEnd, uint _tokenRate, address _tokenAddress, uint _fundingGoal, address _extractor) public {

      require(_icoStart != 0 &&
      _icoEnd != 0 &&
      _icoStart < _icoEnd &&
      _tokenRate != 0 &&
      _tokenAddress != address(0) &&
      _fundingGoal != 0);
      icoStart = _icoStart;
      icoEnd = _icoEnd;
      tokenRate = _tokenRate;
      token = OurToken(_tokenAddress);
      fundingGoal = _fundingGoal;
      owner = msg.sender;
      extractor = _extractor;

   }

   function () public payable {
      buy();
   }

   function buy() public payable {

      require(msg.sender!=owner);
      require(tokensRaised < fundingGoal);
      require(now < icoEnd && now > icoStart);
      uint tokensToBuy;
      uint etherUsed = msg.value;
      tokensToBuy = (etherUsed/ 1 ether) * tokenRate;

      if(tokensRaised + tokensToBuy > fundingGoal) {
         uint exceedingTokens = tokensRaised + tokensToBuy - fundingGoal;
         uint exceedingEther;

         exceedingEther = (exceedingTokens * 1 ether) / tokenRate;
         msg.sender.transfer(exceedingEther);

         tokensToBuy -= exceedingTokens;
         etherUsed -= exceedingEther;
      }

      token.releaseTokens(msg.sender, tokensToBuy);


      tokensRaised += tokensToBuy;
      etherRaised += etherUsed;
   }

   function returnEther() public whenIcoCompleted {

       require(tokensRaised < fundingGoal);
       uint balance = token.balanceOf(msg.sender);
       uint etherToBeReturned = (balance / tokenRate) * 1 ether;
       msg.sender.transfer(etherToBeReturned);
       token.refundTokens(msg.sender,balance);
       etherRaised -= etherToBeReturned;

   }

   function extractEther() public whenIcoCompleted onlyExtractor {
      extractor.transfer(address(this).balance);
   }

 }

Yes, I set the ico address and end time in erc20 token after the ico is deployed.

The contract is accepting ether through payable method when the call to erc20 in the method is commented, but does not work otherwise. What am I doing wrong will calling it?

It is working with Javascript VM but not when tested through ropsten!

Can someone explain it?

Google Structured Data testing tool doesn’t show questions and answers linked together for a Q/A website with…

Please take a look at this:

enter image description here

See? Answers and its question are separated. I mean, they are not link to each other. Also when you open one of the answers, you will see this warning:

enter image description here

Which clearly says (as far as I understand), answers should be a property of a the question.


Ok, I’ve a Q/A website and implemented schema for it this way

enter image description here

See? You don’t see any answer item there (noted that question has 5 answers) and all answers are a property of the question. If you click on the “Question” tab you can see the question with along all its answers.


Now I want to know, I did it right or I should make it like how SO did it?

What relates to unit testing as requirements relate to system testing?

In system testing, a software system is tested against requirements.

In unit testing, a software module is tested against the internal requirements for this module which depend on the specific software design and are not visible to the users in general.

I don’t like the term internal requirement because a requirement is normally something that is visible to the users of the software and does not depend on its internal structure.

Is there an unambiguous and commonly used term for these internal requirements?