Remove slash / delimiters and add the Dart delimiters : r'^[a-z ,.'-]+$ '

Create a static final field for the RegExp to avoid creating a new instance every time a value is checked. Creating a RegExp is expensive.

static final RegExp nameRegExp = RegExp('[a-zA-Z]'); 
    // or RegExp(r'\p{L}'); // see https://stackoverflow.com/questions/3617797/regex-to-match-only-letters 
static final RegExp numberRegExp = RegExp(r'\d');

Then use it like

validator: (value) => value.isEmpty 
    ? 'Enter Your Name'
    : (nameRegExp.hasMatch(value) 
        ? null 
        : 'Enter a Valid Name');

Since WhitelistingTextInputFormatter is deprecated in Flutter as of 1.20, FilteringTextInputFormatter can be used:

A TextInputFormatter that prevents the insertion of characters matching (or not matching) a particular pattern.

Instances of filtered characters found in the new TextEditingValues will be replaced with the replacementString which defaults to the empty string.

Since this formatter only removes characters from the text, it attempts to preserve the existing TextEditingValue.selection to values it would now fall at with the removed characters.

Example use:

TextField(
 keyboardType: TextInputType.numberWithOptions(decimal: true),
 inputFormatters: <TextInputFormatter>[
      FilteringTextInputFormatter.allow(RegExp(r'^\d+(?:\.\d+)?$')),
  ], 
),

Legacy answer

WhitelistingTextInputFormatter "creates a formatter that allows only the insertion of whitelisted characters patterns". It means your pattern should just match any 1 digit or 1 dot.

Also, you need to use a raw string literal if you want to use a single backslash with regex escapes.

Use

WhitelistingTextInputFormatter(RegExp(r"[\d.]"))

Note that if you want to validate the whole input sequence you need to define a validator: validateMyInput and then add

String validateMyInput(String value) {
    Pattern pattern = r'^\d+(?:\.\d+)?$';
    RegExp regex = new RegExp(pattern);
    if (!regex.hasMatch(value))
      return 'Enter Valid Number';
    else
      return null;
  }

Adapted from Form Validation in Flutter.